Cannot Read HM-10 After Connecting with Bluez - raspberry-pi

Question

I am hoping you can help me figure out how to use Bluez to read from an HM-10 BLE module. Why is this a problem for me? I cannot get read or write to work correctly, and I my end-goal is to use Ian Harvey's bluepy library, built on top Bluez stack. Any help is appreciated. Thank you!
The HM-10, connected to an arduino nano, will say "Foobar" , wait one second, say "Barfoo", wait one second, and repeat.
My iOS Bluetooth Serial App (named "Serial" on app store) picks this up correctly.
Hardware: Raspberry Pi ZeroW
Kernel: 4.9.68+
Bluez: 5.50 (released 3 June 2018)
Main Problem
I am unable to use bluetoothctl to read "Foobar" and "Barfoo."
pi#raspberrypi:~ $ bluetoothctl
[bluetooth]# power on
[bluetooth]# connect 34:15:13:87:98:37
[DSDTECH HM-10]# menu gatt
[DSDTECH HM-10]# select-attribute 0000ffe1-0000-1000-8000-00805f9b34fb
[DSDTECH HM-10:/service0010/char0011]# read
[CHG] Attribute /org/bluez/hci0/dev_34_15_13_87_98_37/service0010/char0011 Value:
34 15 13 87 98 37 4....7
34 15 13 87 98 37 4....7
BtMon shows that I am receiving "Foobar" and "Barfoo," though
In fact, using btmon ( $sudo btmon in another terminal), I can see that my raspberry pi zero W is seeing these values. The below entries repeat every other second.
> ACL Data RX: Handle 64 flags 0x02 dlen 15 #278 [hci0] 339.880027
ATT: Handle Value Notification (0x1b) len 10
Handle: 0x0012
Data: 426172666f6f0d0a
> ACL Data RX: Handle 64 flags 0x02 dlen 15 #279 [hci0] 340.292455
ATT: Handle Value Notification (0x1b) len 10
Handle: 0x0012
Data: 466f6f6261720d0a
426172666f6f0d0a = Barfoo (hex2ascii)
466f6f6261720d0a = Foobar (hex2ascii)
Not getting readings from the other attributes
If you know HM-10, you know that ffe1 is what you are supposed to use for uart data transfer. I encoded ffe1 in my homebrewed android application, which works as a master to pair with the HM-10. But, I wanted to check what the other attributes give me.
[DSDTECH HM-10:/service0010/char0011]# select-attribute 00002902-0000-1000-8000-00805f9b34fb
[DSDTECH HM-10:/service0010/char0011/desc0013]# read
[CHG] Attribute /org/bluez/hci0/dev_34_15_13_87_98_37/service0010/char0011/desc0013 Value:
00 00 ..
00 00 ..
[DSDTECH HM-10:/service0010/char0011/desc0013]# list-attributes
Primary Service
/org/bluez/hci0/dev_34_15_13_87_98_37/service000c
00001801-0000-1000-8000-00805f9b34fb
Generic Attribute Profile
Characteristic
/org/bluez/hci0/dev_34_15_13_87_98_37/service000c/char000d
00002a05-0000-1000-8000-00805f9b34fb
Service Changed
Descriptor
/org/bluez/hci0/dev_34_15_13_87_98_37/service000c/char000d/desc000f
00002902-0000-1000-8000-00805f9b34fb
Client Characteristic Configuration
Primary Service
/org/bluez/hci0/dev_34_15_13_87_98_37/service0010
0000ffe0-0000-1000-8000-00805f9b34fb
Unknown
Characteristic
/org/bluez/hci0/dev_34_15_13_87_98_37/service0010/char0011
0000ffe1-0000-1000-8000-00805f9b34fb
Unknown
Descriptor
/org/bluez/hci0/dev_34_15_13_87_98_37/service0010/char0011/desc0013
00002902-0000-1000-8000-00805f9b34fb
Client Characteristic Configuration
Descriptor
/org/bluez/hci0/dev_34_15_13_87_98_37/service0010/char0011/desc0014
00002901-0000-1000-8000-00805f9b34fb
[DSDTECH HM-10:/service0010/char0011/desc0013]# select-attribute /org/bluez/hci0/dev_34_15_13_87_98_37/service0010/char0011/desc0014
# I could not get the uuid to work after selecting 00002902, and do not know how to exit out of an attribute.
# This corresponds to uuid 00002901
[DSDTECH HM-10:/service0010/char0011/desc0014]# read
[CHG] Attribute /org/bluez/hci0/dev_34_15_13_87_98_37/service0010/char0011/desc0014 Value:
77 77 77 2e 6a 6e 68 75 61 6d 61 6f 2e 63 6e www.jnhuamao.cn
77 77 77 2e 6a 6e 68 75 61 6d 61 6f 2e 63 6e www.jnhuamao.cn
[DSDTECH HM-10:/service0010/char0011/desc0014]# select-attribute /org/bluez/hci0/dev_34_15_13_87_98_37/service0010
[DSDTECH HM-10:/service0010]# read
Unable to read attribute /org/bluez/hci0/dev_34_15_13_87_98_37/service0010
[DSDTECH HM-10:/service0010]# select-attribute /org/bluez/hci0/dev_34_15_13_87_98_37/service000c/char000d
# This corresponds to uuid 00002a05-0000-1000-8000-00805f9b34fb
[DSDTECH HM-10:/service000c/char000d]# read
Failed to read: org.bluez.Error.NotPermitted
[DSDTECH HM-10:/service000c/char000d]# select-attribute /org/bluez/hci0/dev_34_15_13_87_98_37/service000c/char000d/desc000f
[CHG] Attribute /org/bluez/hci0/dev_34_15_13_87_98_37/service000c/char000d/desc000f Value:
02 00 ..
02 00 ..
I seem to be reading manufacturer data value
Interestingly, this is the same response as manufacturer data vale, as seen with the info command.
[bluetooth]# info 34:15:13:87:98:37
Device 34:15:13:87:98:37 (public)
Name: DSDTECH HM-10
Alias: DSDTECH HM-10
Paired: no
Trusted: yes
Blocked: no
Connected: no
LegacyPairing: no
UUID: Unknown (0000ffe0-0000-1000-8000-00805f9b34fb)
ManufacturerData Key: 0x4d48
ManufacturerData Value:
34 15 13 87 98 37 4....7
ServiceData Key: 0000b000-0000-1000-8000-00805f9b34fb
ServiceData Value:
00 00 00 00 ....
RSSI: -56
TxPower: 0
When I use a different nano/HM-10 combo, this value is a different gibberish of #|...x , both for the info/manufacturing data and for the read of uuid ffe1.
Additionally, when I initially connect to the device, btmon shows
# MGMT Event: Device Found (0x0012) plen 57 {0x0001} [hci0] 67.415544
LE Address: 34:15:13:87:98:37 (OUI 34-15-13)
RSSI: -62 dBm (0xc2)
Flags: 0x00000000
Data length: 43
Flags: 0x06
LE General Discoverable Mode
BR/EDR Not Supported
Company: not assigned (19784)
Data: 341513879837
Service Data (UUID 0xb000): 00000000
16-bit Service UUIDs (partial): 1 entry
Unknown (0xffe0)
TX power: 0 dBm
Name (complete): DSDTECH HM-10
341513879837 = 47 (hex2ascii) (there are four squares, which I cannot get to display properly. These line up with the four periods)

Answer 1

btmon logs show that the data coming from your HM-10 device are in the form of notifications. In BLE you have three ways to data transfer; read, write and notify. Read is that a GATT client (in your case Bluez) reads data from a service characteristic or from service descriptor of GATT server, provided that read operation is permitted on those respective characteristic or descriptor. Write operation is that a GATT client sends data to GATT server (HM-10) by writing to its service characteristic or descriptor.
The only way by which GATT server sends data to GATT client by itself is by notifications. But the GATT client need to enable notifications on GATT server after establishing connection. Enabling notification can be done by writing to Client Characteristic Configuration Descriptors (CCC) of GATT server. CCC is a special GATT service defined in Bluetooth Core specs.
Once notifications are enabled you will see data from GATT server. With Bluez you can do all BLE operations using gatttool. Below is a example:
For example if the Bluetooth device address of HM-10 is 03:0F:45:65:43:FF and your device hci interface address is hci0, below sequence of commands enables notifications:
gatttool -i hci0 -b 03:0F:45:65:43:FF -I
[03:0F:45:65:43:FF][LE]>connect
Attempting to connect to 03:0F:45:65:43:FF
Connection successful
# lists all other primary services
[03:0F:45:65:43:FF][LE]> primary
attr handle: 0x0001, end grp handle: 0x0005 uuid: 00001800-0000-1000-8000-00805f9b34fb
attr handle: 0x0006, end grp handle: 0x0009 uuid: 00001801-0000-1000-8000-00805f9b34fb
# lists all characteristics
[03:0F:45:65:43:FF][LE]> characteristics
handle: 0x0002, char properties: 0x02, char value handle: 0x0003, uuid: 00002a00-0000-1000-8000-00805f9b34fb
handle: 0x0004, char properties: 0x02, char value handle: 0x0005, uuid: 00002a01-0000-1000-8000-00805f9b34fb
# 2902 is UUID of CCC service
[03:0F:45:65:43:FF][LE]> char-read-uuid 2902
handle: 0x0009 value: 00 00
handle: 0x0019 value: 00 00
# Enable notifications
[03:0F:45:65:43:FF][LE]> char-write-req 0x0009 0100
Characteristic value was written successfully
[03:0F:45:65:43:FF][LE]> char-write-req 0x0019 0100
Characteristic value was written successfully

Answer 2

I know nothing about Bluez, but what you receive is the advertising data, and not the Tx data from the HM10.
To read the data, you must use the service: "0000ffe0-0000-1000-8000-00805f9b34fb" and the
characteristic: "0000ffe1-0000-1000-8000-00805f9b34fb".
You must write the notification descriptor to this characteristic.
To get the data you don't have to do a read, the data is in the notification-payload and you read it in the characteristic-changed event,
or what this is called in Bleuez.