Out-of-order / scrambled samples with lib_tsn AN00203

[akp]

Hello,

I am getting a problem with lib_tsn AN00203 where I occasionally get scrambled data -- actually it appears to be out-of-order -- primarily on streams 3 and 4 in the 32in/32out build. Here are the steps I used to generate the problem:

1. I started with lib_tsn-8.0.0rc1 retrieved from https://github.com/xmos/lib_tsn/tree/v8.0.0rc1

2. I changed the synthesized sine wave from channels 1&2 to synth on channel 24 (last channel of stream 3) and channel 25 (first channel of stream 4) with the following code at line 454 of main.xc of AN00203 (this is the only code change from the example):

Code: Select all

    on tile[0]: [[distribute]] buffer_manager_to_tdm(i_tdm, c_audio, i_i2c[I2S_TO_I2C], c_sound_activity, 0x01800000,
                                                     i_gpio[0], i_gpio[1], i_gpio[2], i_gpio[3]);

3. Build using xTIMEcomposer Community 14.3.0. Allow IDE to download all required modules, except lib_random-master from https://github.com/xmos/lib_random

4. Flash xCORE-200 MC Audio Board with binary

5. Connect MC audio board to MacBook running MacOS El-Capitan (10.11.6 at present). I kept the media clock at Mac System Clock (INPUT_STREAM_DERIVED) but I think the problem may show up more frequently using MC audio board LOCAL_CLOCK. The MacBook Ethernet port is set to DHCP and connects 1000baseT.

6. Start recording with Audacity. I used 25 ch recording and used display Spectrogram to easily see the spurs caused by the scrambling. If ch 24 and 25 are both clean, Cmd-Q quit audacity, cycle power on MC Audio board, and repeat test.

On my 7th attempt I got the attached result, scrambling on ch 25. If you look at the attached screen shots (one showing complete periods of the sine tone, the other zoomed in to show detail), you can see the bad samples come in every 6th sample. What it looks like to me is that the samples are actually coming in out of order (e.g. samples 6,1,2,3,4,5 rather than 1,2,3,4,5,6). That would make sense since the AVB packet rate is 8kHz so there would be 6 samples in every AVB packet.

My other tests I have done show that all channels in a stream are scrambled the same (i.e. chs 25-32 would all be scrambled identically, but not necessarily scrambled the same as the channels in the other streams).

Has anyone else experienced this problem and is there a solution for it?

[akp]

The scrambling also occurs with xTIMEcomposer 14.3.2

[astewart]

Hi akp,

Do you ever observe the scrambling with the sine wave on channels 1 and 2 (i.e. the unmodified example)?

--Aaron

[akp]

Aaron,

Have not yet but I will try it. The reason I selected streams 3 and 4 for the synthesized sine waves is because my heavily modified code usually shows the scrambling on stream 3 and/or 4, but obviously XMOS would never look at that code. So I thought the minimum change to show the issue would probably be in changing the channel mask constant from 0x3 to something else with only 2 bits set.

I can tell you that all channels in a stream are scrambled identically so a more useful test would be to synth on channels 1, 9, 16, 25 or something (since that would hit all 4 streams). Synthing on channels 1 and 2 is redundant since they are on the same stream.

Adam

[akp]

Hello,

I found where the data is getting scrambled. It's in avb_1722_talker_send_packets(). When the data gets written to the output packet it doesn't check if the buffer already contains a packet ready to send for that stream. So the first sample(s) that should be in the following packet overwrite the first sample(s) in the queued packet. I have a fix that prevents the scrambling. I think there may be an underlying race condition (possibly related to the timing of configuring a talker stream and then starting it). But regardless the code change fixes the scrambling issue. I would suggest you merge the changed code into the github master (or an equivalent change that also fixes the issue). If anyone can fix the underlying race condition (or whatever is the issue) that would also be good I think.

The original code is here, you can see that it doesn't check if the tx buffer is empty before data gets copied to it (by the avb1722_create_packet() function). The other issue is that the function doesn't run at all if p_buffer->data_ready is 0, even if one or more of the streams has a packet queued for transmit.

Code: Select all

unsafe void avb_1722_talker_send_packets(streaming chanend c_eth_tx_hp,
                                        avb_1722_talker_state_t &st,
                                        ptp_time_info_mod64 &timeInfo,
                                        audio_double_buffer_t &sample_buffer)
{
  volatile audio_double_buffer_t *unsafe p_buffer =  (audio_double_buffer_t *unsafe) &sample_buffer;
  if (!p_buffer->data_ready) {
    return;
  }

  unsigned rd_buf = !p_buffer->active_buffer;
  audio_frame_t * unsafe frame = (audio_frame_t *)&p_buffer->buffer[rd_buf];

  if (st.max_active_avb_stream != -1) {
    for (int i=0; i < (st.max_active_avb_stream+1); i++) {
      if (st.talker_streams[i].active==2) { // TODO: Replace int with enum
        int packet_size = avb1722_create_packet((st.tx_buf[i], unsigned char[]),
                                                st.talker_streams[i],
                                                timeInfo,
                                                frame, i);
        if (!st.tx_buf_fill_size[i]) st.tx_buf_fill_size[i] = packet_size;
      }
      if (i == st.max_active_avb_stream) {
        p_buffer->data_ready = 0;
      }
    }

    for (int i=0; i < (st.max_active_avb_stream+1); i++) {
      int packet_size = st.tx_buf_fill_size[i];
      if (packet_size) {
        ethernet_send_hp_packet(c_eth_tx_hp, &(st.tx_buf[i], unsigned char[])[2], packet_size, ETHERNET_ALL_INTERFACES);
        st.tx_buf_fill_size[i] = 0;
        st.counters.sent_1722++;
        break;
      }
    }
  }
}

I made two changes:
1. Function checks if the tx buffer is ready to be sent before writing to it, and if so, then sends the packet.
2. Even if there is no new audio data ready, the function still runs and checks to see if any of the streams has a queued packet, and then it sends the packet (so when it gets no audio data it doesn't have to rely on the previous fix).

Code: Select all

unsafe void avb_1722_talker_send_packets(streaming chanend c_eth_tx_hp,
                                        avb_1722_talker_state_t &st,
                                        ptp_time_info_mod64 &timeInfo,
                                        audio_double_buffer_t &sample_buffer)
{
  volatile audio_double_buffer_t *unsafe p_buffer =  (audio_double_buffer_t *unsafe) &sample_buffer;

  if (st.max_active_avb_stream != -1) {
    if (p_buffer->data_ready) {

      unsigned rd_buf = !p_buffer->active_buffer;
      audio_frame_t * unsafe frame = (audio_frame_t *)&p_buffer->buffer[rd_buf];

      for (int i=0; i < (st.max_active_avb_stream+1); i++) {
        if (st.talker_streams[i].active==2) { // TODO: Replace int with enum
          int packet_size = st.tx_buf_fill_size[i];
          if (packet_size) { // Buffer contains queued packet -- send immediately before overwriting data
            ethernet_send_hp_packet(c_eth_tx_hp, &(st.tx_buf[i], unsigned char[])[2], packet_size, ETHERNET_ALL_INTERFACES);
            st.tx_buf_fill_size[i] = 0;
            st.counters.sent_1722++;
          }
          packet_size = avb1722_create_packet((st.tx_buf[i], unsigned char[]),
                                                    st.talker_streams[i],
                                                    timeInfo,
                                                    frame, i);
          if (!st.tx_buf_fill_size[i]) st.tx_buf_fill_size[i] = packet_size;
        }
        if (i == st.max_active_avb_stream) {
          p_buffer->data_ready = 0;
        }
      }
    }

    // Flush queued buffers (one per call to function)
    for (int i=0; i < (st.max_active_avb_stream+1); i++) {
      int packet_size = st.tx_buf_fill_size[i];
      if (packet_size) {
        ethernet_send_hp_packet(c_eth_tx_hp, &(st.tx_buf[i], unsigned char[])[2], packet_size, ETHERNET_ALL_INTERFACES);
        st.tx_buf_fill_size[i] = 0;
        st.counters.sent_1722++;
        break;
      }
    }
  }
}

[akp]

I generated a pull request with my fixed code. Maybe the XMOS people have a better fix. https://github.com/xmos/lib_tsn/pull/35

[nippoo]

Hey @akp, sorry for resurrecting a really old thread but I'm glad I found this as I had the same issue - your pull request still hasn't been merged! Is anyone maintaining the lib_tsn library?! It seems to be completely unmaintained which is a shame.

I'm also having a very similar issue on the received streams 3 and 4 (from the example demo code) - streams 1 and 2 are output correctly, stream 3 appears to have a weird timing bug where some channels work and others only work half the time, and stream 4 appears to be offset by one MCLK causing, basically, full-scale noise! Don't know if you ever came across this or if you were building a talker-only device.

[akp]

No worries. If you are using lib_tsn you will have to maintain it yourself. I and others have posted up about several bug fixes you might want to search on and implement.

I did a talker only device so I don't have much I can provide for help on listener streams.

Do you need to use four streams? If you can reduce to a single 32 ch (or however many channela you need) stream there will be less overhead and it might work better.

I would try that first. It will take some code changes but it was a huge improvement for me

[nippoo]

Thanks! I'm thinking it might be worth making and maintaining a fork of it of nobody else is going to do it...

Ah this is a good shout. I absolutely don't need 8-channel streams. I could do 32-channel streams as I'm trying to drive blocks of 32 outputs from a single sending device. (I had originally planned to use TDM16 for everything but it seems like XMOS support is super buggy for it.)