lquadflash Issues Between XTC Versions Topic is solved

Technical questions regarding the XTC tools and programming with XMOS.
markp
Verified
Member++
Posts: 26
Joined: Thu Jan 10, 2019 6:07 pm

Post by markp »

The repeated calls to fl_startImageAdd() start required the flash sector erases and polls the flash's busy bit until the sectors are erased.
This is why the number of calls required varies from run to run, etc

So I may have reproduced your issue - or I may have found a different "issue".

If I have an extra 7 threads in the test (just doing while(1) busy looping), so 8 threads in total, it fails on the 216 - but always the on the first page write (and subsequent pages). You say it is not always the first page write which fails which makes me wonder if I am seeing something else.
If I remove one of these threads, so 7 in total, it passes on 216 (and 316).

I get the same behavior with fl_connect() and fl_connectToDevice(), so again this is different to what you observe.


Do you have 8 threads running on the tile which is doing the DFU?


If so, a possible workaround (worked for my case) is to call:

set_core_high_priority_on();

before calling

fl_writeImagePage()

and

set_core_high_priority_off();

afterwards

This ensures the DFU thread always get scheduled in one of the 5 repeating thread schedule slots.
View Solution
RitchRock
XCore Addict
Posts: 224
Joined: Tue Jan 17, 2017 9:25 pm

Post by RitchRock »

Hi, Thanks for this. Resource usage is per attached image.

Tile 0 has a control task that gets the data from UART and sends it via an interface transaction to the QSPI storage task on Tile 1. On Tile 1, along with the QSPI task, I do have an I2S task, DSP Processing, DSP Control and DSP Calculation tasks. I'm calling set_core_high_priority_on() for the i2S task - there is a limit to how many times I can call this on a task, correct? I wonder if another task is interfering with DFU, but that doesn't really explain some why it would be alright with the 316 part.
You do not have the required permissions to view the files attached to this post.
markp
Verified
Member++
Posts: 26
Joined: Thu Jan 10, 2019 6:07 pm

Post by markp »

So in our reproduction the 216 fails because the tile is clocked at 500 MHz, and looking with a scope, there is an extra clock cycle emitted at the end of a page write request, so the flash rejects the entire page write.
On the 316, the tile runs at 600 MHz and this extra clock cycle is not emitted.

Either by using a faster device (600MHz), reducing the SPI clock frequency (using a supplied SPI spec) or guaranteeing schedule slots for the flash write thread (using set_core_high_priority_on()) allows the flash thread (and fl_writeImagePage()) to stop the clock (generated from a clock block) in time.

There are 5 round-robin thread schedule slots in the HW and over a period of several iterations of the scheduler, the maximum of 8 threads are shared across these 5 slots to get fair execution time. The API set_core_high_priority_on() guarantees a thread gets allocated one of those 5 slots on every iteration. If 5 threads call this API to reserve a slot, I am not sure what would happen to scheduling the other 3!

See https://www.xmos.com/documentation/XM-0 ... 02030.html for more info.

So in summary, changes to the flash library since Tools 14.4 has made this more sensitive to the combination of a 500 MHz device, plus the additional threads running.
I would use one oif the workarounds
- use a slow SPI clk
- or set_core_high_priority_on() and set_core_high_priority_off() around the call to fl_writeImagePage()
RitchRock
XCore Addict
Posts: 224
Joined: Tue Jan 17, 2017 9:25 pm

Post by RitchRock »

Very helpful - thank you! I will test this. I can stop i2S and DSP during DFU so that I can call set_core_high_priority_on() and set_core_high_priority_off() for fl_writeImagePage().

I'm not sure if this is the cause of my second issue with the upgrade image not booting, however.
markp
Verified
Member++
Posts: 26
Joined: Thu Jan 10, 2019 6:07 pm

Post by markp »

Please do get over the first hurdle and get the DFU writing reliable.

And we can investigate why they do not boot. I shall look into reproducing this although we have tests that pass.
RitchRock
XCore Addict
Posts: 224
Joined: Tue Jan 17, 2017 9:25 pm

Post by RitchRock »

I am happy to report that calling set_core_high_priority_on() during my DFU start transaction and then calling set_core_high_priority_off() during my DFU done interface transaction works when also using flconnect() without a spispec file.

Further, my device is also now restarting with the firmware image active and all CRC showing "GOOD". Curious as to why this also solved the issue here - any thoughts?

Would you recommend I also call set_core_high_priority_on() prior to calling fl_writeData() or even fl_readData()?
markp
Verified
Member++
Posts: 26
Joined: Thu Jan 10, 2019 6:07 pm

Post by markp »

I am still doing some experiments on my side with different devices and tool chain versions. I'd like to understand this fully before we can consider a proper fix to go in a later tools release.

On that note, very strange that I believe earlier you said moving to fl_connectToDevice() makes the the upgrade work (the write API will write and read back every page and verify it), but it fails to boot as I cannot reproduce that behavior.

Was this the case?

I wonder if this is due to different flash devices - I was forced to move to a board with a different device and we find they can all behave subtlety differently when they receive an extra clock edge in a transaction.

In principle it would not do any harm to call set_core_high_priority_on() / set_core_high_priority_off() around all APIs that access the flash - it will guarantee a schedule slot in one of the 5 cycles.

However it will affect scheduling of other threads and depending on what they are doing, they may be impacted. And I think these threads in your system are why originally, you saw the unreliable page write/verifies appearing of different pages each time it was run whereas my simplistic reproduction always failed on the first page.

In all our test frameworks we do not have many/any extra threads doing time-sensitive operations at the same time the DFU flash accesses.
RitchRock
XCore Addict
Posts: 224
Joined: Tue Jan 17, 2017 9:25 pm

Post by RitchRock »

Yes, previously using fl_connectToDevice() with my own spispec showed DFU finishing successfully, but upon device reboot the upgrade image would not load. I uploaded the output from --analyze earlier in this thread. Now with the latest change it's all working. Here is the new output from --analyze:

Code: Select all

Loader xflashReadAll_9_38_upgrade_working.bin:
  Size: 0x15CC bytes (0x573 words)
  CRC: 0xCF85FF5C (GOOD)

  User address: 0x2B000
  Search limit: 0x1FF000

  ID number: 0 (0x0)
  ID string: None

Factory image xflashReadAll_9_38_upgrade_working.bin[0]:
  Offset: 0x15D4
  Image tag: 0xCA7FA11
  Page CRC: 0x8EFDF758 (GOOD)
  Image CRC: 0x18AD49AA (GOOD)

  Version: 0x0
  Features: 0x40
    Switch setup present

  Size: 0x28B88 bytes
  Length: 0x28B88 bytes

  Boot address: 0x0
  Overlay address: 0x0
  Overlay offset: 0x0

  Switch setup table offset: 0x38
  Core table offset: 0x40

  Tools version: 15.3.0
  Image format: 5 (Tools 15.2)

    Switch setup xflashReadAll_9_38_upgrade_working.bin[0]:
      Offset: 0x80
      Size: 0x1F28 bytes (0x7CA words)
      CRC: 0x076D28A3 (GOOD)

  Number of cores: 2

    Core xflashReadAll_9_38_upgrade_working.bin[0][0]:
      Offset: 0x1FA8
      Size: 0x132D0 bytes (0x4CB4 words)
      Channel end: 0x80020002
      CRC: 0xF598FB5D (GOOD)

        SWMEM address: 0x0
        SWMEM offset: 0x0
        SWMEM size: 0x0 bytes

        EXTMEM offset: 0x0
        EXTMEM size: 0x0 bytes

    Core xflashReadAll_9_38_upgrade_working.bin[0][1]:
      Offset: 0x15278
      Size: 0x13910 bytes (0x4E44 words)
      Channel end: 0x80030002
      CRC: 0xBC775778 (GOOD)

        SWMEM address: 0x0
        SWMEM offset: 0x0
        SWMEM size: 0x0 bytes

        EXTMEM offset: 0x0
        EXTMEM size: 0x0 bytes

Upgrade image xflashReadAll_9_38_upgrade_working.bin[1]:
  Offset: 0x2B000
  Image tag: 0xCA7FA11
  Page CRC: 0x4A296B61 (GOOD)
  Image CRC: 0xCE82E2C4 (GOOD)

  Version: 0x1
  Features: 0x40
    Switch setup present

  Size: 0x28B88 bytes
  Length: 0x28B88 bytes

  Boot address: 0x0
  Overlay address: 0x0
  Overlay offset: 0x0

  Switch setup table offset: 0x38
  Core table offset: 0x40

  Tools version: 15.3.0
  Image format: 5 (Tools 15.2)

    Switch setup xflashReadAll_9_38_upgrade_working.bin[1]:
      Offset: 0x80
      Size: 0x1F28 bytes (0x7CA words)
      CRC: 0x076D28A3 (GOOD)

  Number of cores: 2

    Core xflashReadAll_9_38_upgrade_working.bin[1][0]:
      Offset: 0x1FA8
      Size: 0x132D0 bytes (0x4CB4 words)
      Channel end: 0x80020002
      CRC: 0xF598FB5D (GOOD)

        SWMEM address: 0x0
        SWMEM offset: 0x0
        SWMEM size: 0x0 bytes

        EXTMEM offset: 0x0
        EXTMEM size: 0x0 bytes

    Core xflashReadAll_9_38_upgrade_working.bin[1][1]:
      Offset: 0x15278
      Size: 0x13910 bytes (0x4E44 words)
      Channel end: 0x80030002
      CRC: 0x988E755D (GOOD)

        SWMEM address: 0x0
        SWMEM offset: 0x0
        SWMEM size: 0x0 bytes

        EXTMEM offset: 0x0
        EXTMEM size: 0x0 bytes

Dumping stage 2 loader to loader.bin
Dumping image 0 to image0.bin
Dumping image 0 switch setup code to image0_swsetup.bin
Dumping image 0 core 0 code to image0_core0.bin
Dumping image 0 core 1 code to image0_core1.bin
Dumping image 1 to image1.bin
Dumping image 1 switch setup code to image1_swsetup.bin
Dumping image 1 core 0 code to image1_core0.bin
Dumping image 1 core 1 code to image1_core1.bin
Regarding my other threads, I did reorganize and make sure I only have 2 of these threads calling set_core_high_priority_on() now per tile at any time. Everything seems to be working as it should.

I'm happy to help as needed if you would like me to run anything on my boards to help with fully understand the issue.
markp
Verified
Member++
Posts: 26
Joined: Thu Jan 10, 2019 6:07 pm

Post by markp »

So an update following some further investigations

1) This is a regression since Tools 14 and is present in XTC 15.1.4, XTC 15.2.1 and XTC 15.3.0
2) It affects ALL fl_...() APIs such as fl_writeImagePage(), fl_getFactoryImage() etc
3) It occurs on devices running at 500 MHz - such as the 216 - but does not on faster 600MHz devices such as the 316. (If the 316 is clocked at 500 MHz it exhibits the problem too)
4) Workarounds are
a) reduce number of threads running when fl_...() APIs are being called
b) reduce flash clock frequency by supplying a SPI-spec and using fl_connectToDevice() - the 4th entry in the SPI spec is the log2 clock div and should be >= 4
c) Always call set_core_high_priority_on() before calling any fl_...() API.
It is wise to call set_core_high_priority_off() afterwards otherwise other system issues may be observed because other threads do not get their necessary scheduling allocation
RitchRock
XCore Addict
Posts: 224
Joined: Tue Jan 17, 2017 9:25 pm

Post by RitchRock »

Thank you for the update and summary. Everything is still working on my end using fl_connect(qspi_ports) and calling set_core_high_priority_on() before any API call (along with set_core_high_priority_off() afterwards).

To be perfectly clear, the workaround mentioned and that I tested are recommended as either option a, b or c individually - not all 3 combined, correct?