Bug: Object reference not set to an instance of an object

Hello!

I have been getting this intermittent error when using the QA401H on both linux (raspberry pi) and macOS. I think I finally found a good repro today.

this is the output of QA401H:

---
Time 09:19:09.334
POST	Request:http://localhost:9401/Acquisition
Acquisition started. Fs: 48000 FFTSize: 32768 Resolution: 1.465Hz Window: Hann Rounding: True  Atten Enabled: False
Exception occurred in DoAcquisition(): Object reference not set to an instance of an object.
Return: { "SessionId":"0" }
Status: 200 (Elapsed: 11.22 mS)
---
Time 09:19:09.351
GET	Request:http://localhost:9401/RmsDbv/20/20000
Bad Request: Exception: (Object reference not set to an instance of an object.)
Status: 400 (Elapsed: 4.63 mS)

note that the exception seems to occur already in the acquisition step, but the acquisition returns OK.

repro:

  • plug in QA401
  • start the QA401H process
  • QA401H configures the QA401 FPGA
  • PUT Request:http://localhost:9401/Settings/Default
  • GET Request:http://localhost:9401/Status/Version
  • PUT Request:http://localhost:9401/Settings/Input/Max/6
  • POST Request:http://localhost:9401/Acquisition
  • QA401H logs error message (see above) but returns success
  • GET Request:http://localhost:9401/RmsDbv/20/20000
  • QA401H returns 400, see above
  • restart QA401H process
  • QA401H does not reconfigure FPGA
  • run same commands again
  • everything runs as expected
Here is the full output of the steps outlined above:

publish | dotnet QA401H.dll
Server started listening on port 9401
Version: 1.0
Loading FX2 with file 16C04E27.hex [b58731ad] (Match)
Usb re-opened
Loading FPGA File: qa401.bit [eeba84b2] (Match)
FPGA File DateTime: 04/04/2019 11:32

Done loading FPGA
===USB DESCRIPTORS===
Endpoint: 0x1 Max Packet Size: 64 bytes
Endpoint: 0x81 Max Packet Size: 64 bytes
Endpoint: 0x2 Max Packet Size: 512 bytes
Endpoint: 0x4 Max Packet Size: 512 bytes
Endpoint: 0x86 Max Packet Size: 512 bytes
Endpoint: 0x88 Max Packet Size: 512 bytes
Waiting for HTTP…
From your web browser, go to http://localhost:9401
Press ‘X’ to exit


Time 09:31:23.47
PUT Request:http://localhost:9401/Settings/Default



Return: { “SessionId”:“0” }
Status: 200 (Elapsed: 6.85 mS)

Time 09:31:23.60
GET Request:http://localhost:9401/Status/Version
Return: { “SessionId”:“0”, “Value”:“1” }
Status: 200 (Elapsed: 1.45 mS)

Time 09:31:28.966
PUT Request:http://localhost:9401/Settings/Input/Max/6
Return: { “SessionId”:“0” }
Status: 200 (Elapsed: 6.27 mS)

Time 09:31:28.980
POST Request:http://localhost:9401/Acquisition
Acquisition started. Fs: 48000 FFTSize: 32768 Resolution: 1.465Hz Window: Hann Rounding: True Atten Enabled: False
Exception occurred in DoAcquisition(): Object reference not set to an instance of an object.
Return: { “SessionId”:“0” }
Status: 200 (Elapsed: 11.67 mS)

Time 09:31:28.998
GET Request:http://localhost:9401/RmsDbv/20/20000
Bad Request: Exception: (Object reference not set to an instance of an object.)
Status: 400 (Elapsed: 7.17 mS)

publish | dotnet QA401H.dll # NOTE: process manually restarted here!
Server started listening on port 9401
Version: 1.0
Usb is already configured
Waiting for HTTP…
From your web browser, go to http://localhost:9401
Press ‘X’ to exit


Time 09:36:31.743
PUT Request:http://localhost:9401/Settings/Default



Return: { “SessionId”:“0” }
Status: 200 (Elapsed: 7.00 mS)

Time 09:36:31.758
GET Request:http://localhost:9401/Status/Version
Return: { “SessionId”:“0”, “Value”:“1” }
Status: 200 (Elapsed: 1.29 mS)

Time 09:36:38.85
PUT Request:http://localhost:9401/Settings/Input/Max/6
Return: { “SessionId”:“0” }
Status: 200 (Elapsed: 7.72 mS)

Time 09:36:38.100
POST Request:http://localhost:9401/Acquisition
Acquisition started. Fs: 48000 FFTSize: 32768 Resolution: 1.465Hz Window: Hann Rounding: True Atten Enabled: False
Acquisition completed. Elapsed 0.78 seconds.
Return: { “SessionId”:“3200886713” }
Status: 200 (Elapsed: 784.94 mS)

Time 09:36:38.892
GET Request:http://localhost:9401/RmsDbv/20/20000
Return: { “SessionId”:“3200886713”, “Left”:“4.580997138893466”, “Right”:“4.6104584993685” }
Status: 200 (Elapsed: 2.43 mS)

Setup info:

  • QA401 version: 1.0 but the bug has been present in previous versions as well.
  • macOS version: Big Sur (11.6) but has been seen on another mac version as well.
  • Linux version: Raspberry Pi OS which is based on Debian

Since it’s been seen on multiple OSes and versions, I would guess it is not related to the OS version but unix in general.

Hope this helps!

Philip

Hi @philipte, just to check…you are using dotnet rather than mono, is that right? (eg launch with dotnet qa401h.dll). This doesn’t repro on Windows, but we’ll setup a fresh ubuntu to try. Thanks for a very succinct set of steps to force this.

Hi @matt!

Yes, I am using dotnet. I set it up using these instructions: https://quantasylum.com/blogs/news/qa401-headless-linux

OK, thanks, I think can repro this in virtualbox with Ubuntu 20.04 LTS.

There are two important events to note. First is after the FPGA configuration completes (dots shown on screen) all the LEDs will momentarily light up. That signals the FPGA accepted the configuration. Next, when the QA401H application is running but idle with the QA401 connected, the LINK LED will remain lit. The application occasionally writes to the FPGA, which keeps the LED on for another 500 mS or so.

In the failed test case above that you shared, do you see all the LEDs light briefly after FPGA config and do you see the LINK LED remain on while the app is awaiting input?

thanks for looking into this!

yes, the three leds light up as they should.

aha! the LINK led is not always lit. the bug only seem to occur when the link led is off.

I guess this could be a clue as well:

  • if I restart QA401H and it does NOT load the FPGA configurations the link led will lights up and everything works as it should.
  • if the FPGA configuration is loaded the link LED does not light up afterwards, and the bug appears.

OK, this is very good feedback, please standby 24 hours. it’s different than the VM case but I think more explained.

hi @matt, did you manage to reproduce on your end?

Hi @philipte, I cannot fully reproduce under VM, but I think your notes on LED behavior indicate consistent successful FPGA config. But just before config, the USB connection is closed and re-opened. The way the first open and the second open changed in version 0.999 in order to get MacOS working. But that second open didn’t get the same code path. So, my thinking is first open runs MacOS code, and the second open (needed if configuring FPGA) doesn’t get the MacOS code. And then, when the app is closed and re-open, and the MacOS code runs at first open and things work. So, this release makes both USB opens the same.

Please download zip from this message and let me know what you see when time permits.

QA401H_1.01.zip (803.4 KB)

thanks @matt!

I’ve tested the attached v. 1.01 on both macOS and linux.

  • on macOS it seems to work perfectly, it connects to QA401h every time and the LINK led is always lit.
  • on linux, it does not work. did not manage to get the LINK light up at all.

I can also confirm that the issue is related to the change in v. 0.999 that made stuff work on macOS. I tried rolling back to 0.998 on my linux machine, and that version works much better although it sometimes needs an occasional power cycle.

I’m not so well versed in cross platform stuff, but it seems like we have one way of connecting to the QA401H that works on linux (v. 0.998) and one way which works on macOS (v 1.01). Could it be an idea to detect which platform the code is running at and choose how to connect?

Furthermore, it would be really useful to be able to detect this state of the LINK led and return it in http://localhost:9401/Status/Connection (or another endpoint?). currently that api endpoint returns True even if the LINK led is not lit. I am not sure how complex that is, but would be very useful when trying to mitigate this error.

Hi @philipte there will be another release tomorrow that senses the OS and applies the MACOS fixes just to the mac. I don’t know if that is the root issue on your linux setup or not. But there is more USB debug output coming too, along with the /Status/Connection actually doing a readback from the FPGA to confirm the connection is functioning.

I know it’s not ideal, but the fallback on linux until this is better understood is that you run the qa401h.dll, configure, check the connection, and if not up then kill the process and re-start and that should give the connection.

For some reason, in linux having the device close the USB port without warning, and then the app re-open it isn’t working and it’s not yet clear why.

Attached is 1.02. If this works as expected I will trim down the verbosity and post on github as 1.03.

Changes are:

1.02

  • “Claiming” interfaces now only happens for OSX
  • /Status/Connection does write/read probe of FPGA to verify connection. If not connected, this will take several seconds to fail.
  • Detected OS version is shown on startup.
  • Status dots on FX2 loading (similar to FPGA)

Thanks!

QA401H_1.02.zip (806.5 KB)

Matt, sorry for the late response!

I have tested it on both macOS and Linux, and it can connect to QA401 with link led lit on both platforms. :+1:

did setup for a second linux machine today (identical to the first, debian 10).

at first, I got:

Linux Detected
Version: 1.02
Server started listening on port 9401
Exception in Usb.Open: Object reference not set to an instance of an object.

but after a reboot of the machine it all works fine. :+1:

Thanks for helping out @matt!

Hi @philipte, thanks very much for reporting and helping to sort this out. Release 1.03 is posted below

1 Like

Thanks for the support, @matt!