XUD_DoGetRequest stalls Topic is solved

[dsteinwe]

Hello,

I use the XUD library in the lastest 2.1.0 from https://github.com/xmos/lib_xud/tree/develop/lib_xud inside my usb sound card project. In some circumdances, when I call XUD_DoGetRequest, this function stalls and never returns. The problem arises at endpoint 0, that the sound card controls.

To provoke this error faster, I wrote a shell script under Linux. The script switches a control in a loop between two states. Switching this control causes the sound card to send two interrupts to the host, because the states of two other controls have changed as a result of that change. Due to the interrupts, the host also reads the states of the other two controls. Very rarely, the sound card does not answer one of these host requests.

The implementation for the state request is nothing special: A endpoint 0 handler processes the host requests. The following pseudo code shows the implementation for answering a "get" request:

Code: Select all

// verify, if the request is valid
// load the state from a variable and store it to the buffer
XUD_DoGetRequest(ep0_out, ep0_in, buffer, 1, sp.wLength);

In rare cases, the call of the function XUD_DoGetRequest stalls. I followed up the call:
XUD_DoGetRequest calls the function XUD_SetBuffer_EpMax. This function in turn calls "XUD_SetBuffer" (inside "if (datalength <= epMax) { ... }"). XUD_SetBuffer calls XUD_SetBuffer_Finish and here the error occurs in the following line:

Code: Select all

asm volatile("testct %0, res[%1]" : "=r"(isReset) : "r"(ep->client_chanend));

"testct" causes a wait, that never returns. I have no idea why.

[fabriceo]

hi dsteinwe,
this is not going to help you much, but as you probably know the testct is a blocking instruction if there is no token received in the channel end...
good luck and keep us posted :)

[dsteinwe]

Fabriceo, you're right. I guess that the "testct" inside my ep0 handler thread is waiting for a control token from the XUD_Main thread signaling that the response has been sent to the host. Anyhow, this control token is never sent. The question is: why? In wireshark I got following result:

The request from the host:

Code: Select all

Frame 1707931: 64 bytes on wire (512 bits), 64 bytes captured (512 bits) on interface usbmon1, id 0
    Interface id: 0 (usbmon1)
    Encapsulation type: USB packets with Linux header and padding (115)
    Arrival Time: Feb 20, 2023 11:50:56.873237000 CET
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1676890256.873237000 seconds
    [Time delta from previous captured frame: 0.000008000 seconds]
    [Time delta from previous displayed frame: 0.000165000 seconds]
    [Time since reference or first frame: 33.297026000 seconds]
    Frame Number: 1707931
    Frame Length: 64 bytes (512 bits)
    Capture Length: 64 bytes (512 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: usb]
USB URB
    [Source: host]
    [Destination: 1.15.0]
    URB id: 0xffff9b931b350840
    URB type: URB_SUBMIT ('S')
    URB transfer type: URB_CONTROL (0x02)
    Endpoint: 0x80, Direction: IN
    Device: 15
    URB bus id: 1
    Device setup request: relevant (0)
    Data: not present ('<')
    URB sec: 1676890256
    URB usec: 873237
    URB status: Operation now in progress (-EINPROGRESS) (-115)
    URB length [bytes]: 1
    Data length [bytes]: 0
    [Response in: 1707939]
    Interval: 0
    Start frame: 0
    Copy of Transfer Flags: 0x00000200, Dir IN
    Number of ISO descriptors: 0
    [bInterfaceClass: Audio (0x01)]
Setup Data
    bmRequestType: 0xa1
    bRequest: 1
    wValue: 0x0100
    wIndex: 5120 (0x1400)
    wLength: 1

I have verified the "Setup Data". There are no errors.

The response from the device:

Code: Select all

Frame 1707939: 64 bytes on wire (512 bits), 64 bytes captured (512 bits) on interface usbmon1, id 0
    Interface id: 0 (usbmon1)
    Encapsulation type: USB packets with Linux header and padding (115)
    Arrival Time: Feb 20, 2023 11:50:56.873393000 CET
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1676890256.873393000 seconds
    [Time delta from previous captured frame: 0.000029000 seconds]
    [Time delta from previous displayed frame: 0.000156000 seconds]
    [Time since reference or first frame: 33.297182000 seconds]
    Frame Number: 1707939
    Frame Length: 64 bytes (512 bits)
    Capture Length: 64 bytes (512 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: usb]
USB URB
    [Source: 1.15.0]
    [Destination: host]
    URB id: 0xffff9b931b350840
    URB type: URB_COMPLETE ('C')
    URB transfer type: URB_CONTROL (0x02)
    Endpoint: 0x80, Direction: IN
    Device: 15
    URB bus id: 1
    Device setup request: not relevant ('-')
    Data: present (0)
    URB sec: 1676890256
    URB usec: 873393
    URB status: Protocol error (-EPROTO) (-71)
    URB length [bytes]: 0
    Data length [bytes]: 0
    [Request in: 1707931]
    [Time from request: 0.000156000 seconds]
    Unused Setup Header
    Interval: 0
    Start frame: 0
    Copy of Transfer Flags: 0x00000200, Dir IN
    Number of ISO descriptors: 0
    [bInterfaceClass: Audio (0x01)]

The URB status is set to -71, that means an protocol error has occurred. But why? The request from the host was error-free and my ep0 handler also processes this request without an error.

(update: wireshark snippets fixed)

[dsteinwe]

In the meantime, I've modified some test conditions to better understand the problem. The biggest impact is when I changed the script. In the current version, there is delay of 0.01s between switching the control. The first script version had a delay of 0.3s. I also have tested different frequencies for the input and output to modify the bus load. With the script change, I was able to generate an error at any standard sample frequency from 44100 up to 192000Hz. In very few cases the exception ET_LOAD_STORE does occur. Two files were claimed:

Code: Select all

xrun: Program received signal ET_LOAD_STORE, Memory access exception.
      [Switching to tile[1] core[4] (dual issue)]
      Pid_In () at ./included/XUD_Token_In_DI.S:54

      54	./included/XUD_Token_In_DI.S: No such file or directory.
      	in ./included/XUD_Token_In_DI.S
      Current language:  auto; currently asm

Code: Select all

xrun: Program received signal ET_LOAD_STORE, Memory access exception.
      [Switching to tile[1] core[4] (dual issue)]
      Pid_Setup () at ./included/XUD_Token_Setup_DI.S:15

      15	./included/XUD_Token_Setup_DI.S: No such file or directory.
      	in ./included/XUD_Token_Setup_DI.S
      Current language:  auto; currently asm

My current hypothesis is that the USB protocol error and exceptions are caused by an invalid write memory access. I think that the source is not to be found in the xud library but in my code. Is there a (simple) way to locate such an invalid memory access?

[dsteinwe]

The problem becomes weird: When I comment the line for the output on the other tile, the script never stops. Ok, I had only run it about 15min, but usually the problem occurs within 1min. What does make the difference? Switching the control between the two states, has the effect that clock for the output has to re-synchronize. That means, that the streaming channel for spdif output get filled with data until the channel saturated. I guess, this happens to all channels over all threads and tiles, that processes and transports the output data over streaming channels. But why does this affect to the xud main thread? Puh, I only can guess: The xud main thread is connected to multiple other threads in order of the different endpoints and its corresponding channels. When the channel for the output stream comes to a standstill due to congestion in other threads/channels, the main xud thread cannot process any further requests/responses from other endpoints correctly.

Anyhow, I think the solution is to avoid the congestion. I currently have no idea how I will implement this. Is it possible to query how full the buffer is for channels comparable to ports?

[dsteinwe]

An streaming channel can buffer 8 bytes until it block. I didn't find any asm instruction to check the buffer state of an streaming channel. An port has a transfer and shift register. Anyway, I didn't find any dedicated asm instruction for my problem. Maybe that the instructions "peek" or "getts" can help to solve the problem. So far, I haven't found any useful and robust implementation with these instructions. Ideas are welcome.

Right now, I use a stupid implementation, that skips writing to the output channels for a while when the control is switched. The disadvantage of this implementation is that the output doesn't start immediately, but after 2.5ms. Anyway, it solves my problem at first.

Thanks for listing ;-)