AN00170 "Using the SDRAM library"

If you have a simple question and just want an answer.
User avatar
aneves
Experienced Member
Posts: 93
Joined: Wed Sep 16, 2015 2:38 pm

Post by aneves »

Thanks for the tip.

I added the debugging code where I print out the rowcol variable on the write side. When I write 2,097,188 bytes (under the threshold) I see the following last few lines of output:

Code: Select all

rowcol: 8178
rowcol: 31989747
rowcol: 8180
rowcol: 31989749
rowcol: 8182
rowcol: 31989751
rowcol: 8184
rowcol: 31989753
rowcol: 8186
rowcol: 31989755
rowcol: 8188
rowcol: 31989757
rowcol: 8190
rowcol: 31989759
Not sure if I understand what's going here but it seems to follow a simple pattern.

Now, when I cross that threshold and add an extra byte (2,097,189 bytes total) here is the last few lines of output:

Code: Select all

rowcol: 8182
rowcol: 31989751
rowcol: 8184
rowcol: 31989753
rowcol: 8186
rowcol: 31989755
rowcol: 8188
rowcol: 31989757
rowcol: 8190
rowcol: 31989759
rowcol: 536879104
That last address does not seem like the others. If I add another byte to the write buffer, the rowcol output is identical to this. If I add 100 bytes to the write buffer, I get the identical rowcol output.

I also noticed this in server.xc:

Code: Select all

#define BANK_SHIFT          (13)//FIXME 15 - bank_address_bits
Is the FIXME a clue? Looks to me like an addressing issue but I need your guidance on that.

Thanks!


User avatar
aneves
Experienced Member
Posts: 93
Joined: Wed Sep 16, 2015 2:38 pm

Post by aneves »

Hi Guys,

Any ideas? Does it look like the addressing in the server.xc file is not working as expected?
User avatar
infiniteimprobability
XCore Legend
Posts: 1126
Joined: Thu May 27, 2010 10:08 am
Contact:

Post by infiniteimprobability »

Can you show me the exact set of arguments being passed to:

Code: Select all

sdram_write(c_server, sdram_state, addr, BUF_WORDS, move(write_buffer_pointer));
well in particular addr and BUFF_WORDS. Both of these (slightly confusingly*) are long word values.

I annotated the code (using hex format which is easier to decipher) using the following test

Code: Select all

  for (unsigned addr=(2097152 - 16); addr < (2097188 + 4); addr++){
    printf("**Write %dx 16b words at addr=0x%x (%d)\n", BUF_WORDS*2, addr, addr);
    sdram_write(c_server, sdram_state, addr, BUF_WORDS, move(write_buffer_pointer));
    sdram_complete(c_server, sdram_state, write_buffer_pointer);
  }
and got this:

Code: Select all

Start App
**Write 16x 16b words at addr=0x1ffff0 (2097136)
row=1fff	bank=0	col=1e0	rowcol=1e01fff	word_count=8
**Write 16x 16b words at addr=0x1ffff1 (2097137)
row=1fff	bank=0	col=1e2	rowcol=1e21fff	word_count=8
**Write 16x 16b words at addr=0x1ffff2 (2097138)
row=1fff	bank=0	col=1e4	rowcol=1e41fff	word_count=8
**Write 16x 16b words at addr=0x1ffff3 (2097139)
row=1fff	bank=0	col=1e6	rowcol=1e61fff	word_count=8
**Write 16x 16b words at addr=0x1ffff4 (2097140)
row=1fff	bank=0	col=1e8	rowcol=1e81fff	word_count=8
**Write 16x 16b words at addr=0x1ffff5 (2097141)
row=1fff	bank=0	col=1ea	rowcol=1ea1fff	word_count=8
**Write 16x 16b words at addr=0x1ffff6 (2097142)
row=1fff	bank=0	col=1ec	rowcol=1ec1fff	word_count=8
**Write 16x 16b words at addr=0x1ffff7 (2097143)
row=1fff	bank=0	col=1ee	rowcol=1ee1fff	word_count=8
**Write 16x 16b words at addr=0x1ffff8 (2097144)
row=1fff	bank=0	col=1f0	rowcol=1f01fff	word_count=8
**Write 16x 16b words at addr=0x1ffff9 (2097145)
row=1fff	bank=0	col=1f2	rowcol=1f21fff	word_count=7
row=0	bank=1	col=0	rowcol=20002000	word_count=1
**Write 16x 16b words at addr=0x1ffffa (2097146)
row=1fff	bank=0	col=1f4	rowcol=1f41fff	word_count=6
row=0	bank=1	col=0	rowcol=20002000	word_count=2
**Write 16x 16b words at addr=0x1ffffb (2097147)
row=1fff	bank=0	col=1f6	rowcol=1f61fff	word_count=5
row=0	bank=1	col=0	rowcol=20002000	word_count=3
**Write 16x 16b words at addr=0x1ffffc (2097148)
row=1fff	bank=0	col=1f8	rowcol=1f81fff	word_count=4
row=0	bank=1	col=0	rowcol=20002000	word_count=4
**Write 16x 16b words at addr=0x1ffffd (2097149)
row=1fff	bank=0	col=1fa	rowcol=1fa1fff	word_count=3
row=0	bank=1	col=0	rowcol=20002000	word_count=5
**Write 16x 16b words at addr=0x1ffffe (2097150)
row=1fff	bank=0	col=1fc	rowcol=1fc1fff	word_count=2
row=0	bank=1	col=0	rowcol=20002000	word_count=6
**Write 16x 16b words at addr=0x1fffff (2097151)
row=1fff	bank=0	col=1fe	rowcol=1fe1fff	word_count=1
row=0	bank=1	col=0	rowcol=20002000	word_count=7
**Write 16x 16b words at addr=0x200000 (2097152)
row=0	bank=1	col=0	rowcol=20002000	word_count=8
**Write 16x 16b words at addr=0x200001 (2097153)
row=0	bank=1	col=2	rowcol=20022000	word_count=8
**Write 16x 16b words at addr=0x200002 (2097154)
row=0	bank=1	col=4	rowcol=20042000	word_count=8
**Write 16x 16b words at addr=0x200003 (2097155)
row=0	bank=1	col=6	rowcol=20062000	word_count=8
**Write 16x 16b words at addr=0x200004 (2097156)
row=0	bank=1	col=8	rowcol=20082000	word_count=8
**Write 16x 16b words at addr=0x200005 (2097157)
row=0	bank=1	col=a	rowcol=200a2000	word_count=8
**Write 16x 16b words at addr=0x200006 (2097158)
row=0	bank=1	col=c	rowcol=200c2000	word_count=8
**Write 16x 16b words at addr=0x200007 (2097159)
row=0	bank=1	col=e	rowcol=200e2000	word_count=8
**Write 16x 16b words at addr=0x200008 (2097160)
row=0	bank=1	col=10	rowcol=20102000	word_count=8
**Write 16x 16b words at addr=0x200009 (2097161)
row=0	bank=1	col=12	rowcol=20122000	word_count=8
**Write 16x 16b words at addr=0x20000a (2097162)
row=0	bank=1	col=14	rowcol=20142000	word_count=8
**Write 16x 16b words at addr=0x20000b (2097163)
row=0	bank=1	col=16	rowcol=20162000	word_count=8
**Write 16x 16b words at addr=0x20000c (2097164)
row=0	bank=1	col=18	rowcol=20182000	word_count=8
**Write 16x 16b words at addr=0x20000d (2097165)
row=0	bank=1	col=1a	rowcol=201a2000	word_count=8
**Write 16x 16b words at addr=0x20000e (2097166)
row=0	bank=1	col=1c	rowcol=201c2000	word_count=8
**Write 16x 16b words at addr=0x20000f (2097167)
row=0	bank=1	col=1e	rowcol=201e2000	word_count=8
**Write 16x 16b words at addr=0x200010 (2097168)
row=0	bank=1	col=20	rowcol=20202000	word_count=8
**Write 16x 16b words at addr=0x200011 (2097169)
row=0	bank=1	col=22	rowcol=20222000	word_count=8
**Write 16x 16b words at addr=0x200012 (2097170)
row=0	bank=1	col=24	rowcol=20242000	word_count=8
**Write 16x 16b words at addr=0x200013 (2097171)
row=0	bank=1	col=26	rowcol=20262000	word_count=8
**Write 16x 16b words at addr=0x200014 (2097172)
row=0	bank=1	col=28	rowcol=20282000	word_count=8
**Write 16x 16b words at addr=0x200015 (2097173)
row=0	bank=1	col=2a	rowcol=202a2000	word_count=8
**Write 16x 16b words at addr=0x200016 (2097174)
row=0	bank=1	col=2c	rowcol=202c2000	word_count=8
**Write 16x 16b words at addr=0x200017 (2097175)
row=0	bank=1	col=2e	rowcol=202e2000	word_count=8
**Write 16x 16b words at addr=0x200018 (2097176)
row=0	bank=1	col=30	rowcol=20302000	word_count=8
**Write 16x 16b words at addr=0x200019 (2097177)
row=0	bank=1	col=32	rowcol=20322000	word_count=8
**Write 16x 16b words at addr=0x20001a (2097178)
row=0	bank=1	col=34	rowcol=20342000	word_count=8
**Write 16x 16b words at addr=0x20001b (2097179)
row=0	bank=1	col=36	rowcol=20362000	word_count=8
**Write 16x 16b words at addr=0x20001c (2097180)
row=0	bank=1	col=38	rowcol=20382000	word_count=8
**Write 16x 16b words at addr=0x20001d (2097181)
row=0	bank=1	col=3a	rowcol=203a2000	word_count=8
**Write 16x 16b words at addr=0x20001e (2097182)
row=0	bank=1	col=3c	rowcol=203c2000	word_count=8
**Write 16x 16b words at addr=0x20001f (2097183)
row=0	bank=1	col=3e	rowcol=203e2000	word_count=8
**Write 16x 16b words at addr=0x200020 (2097184)
row=0	bank=1	col=40	rowcol=20402000	word_count=8
**Write 16x 16b words at addr=0x200021 (2097185)
row=0	bank=1	col=42	rowcol=20422000	word_count=8
**Write 16x 16b words at addr=0x200022 (2097186)
row=0	bank=1	col=44	rowcol=20442000	word_count=8
**Write 16x 16b words at addr=0x200023 (2097187)
row=0	bank=1	col=46	rowcol=20462000	word_count=8
**Write 16x 16b words at addr=0x200024 (2097188)
row=0	bank=1	col=48	rowcol=20482000	word_count=8
**Write 16x 16b words at addr=0x200025 (2097189)
row=0	bank=1	col=4a	rowcol=204a2000	word_count=8
**Write 16x 16b words at addr=0x200026 (2097190)
row=0	bank=1	col=4c	rowcol=204c2000	word_count=8
**Write 16x 16b words at addr=0x200027 (2097191)
row=0	bank=1	col=4e	rowcol=204e2000	word_count=8
I'd say the addressing looks fine. The server is calculating the row, bank and col OK. It is also properly splitting the transaction into two writes where it crosses a boundary.

More importantly for this thread, it is passing through the 2,097,188 boundary without doing anything silly.

I think the next thing to check is your usage. Showing what arguments are being passed (and preferably a post of your test code) will be helpful...
User avatar
infiniteimprobability
XCore Legend
Posts: 1126
Joined: Thu May 27, 2010 10:08 am
Contact:

Post by infiniteimprobability »

Sorry - didn't answer your question about:

Code: Select all

#define BANK_SHIFT          (13)//FIXME 15 - bank_address_bits
This shift value looks completely correct. On the SDRAM slice reference schematic, the bank select lines are on DQ13 and D14 so a 13 bit shift gets them to the right place. 15 bits would cause the upper bank line to be pushed into the column address cycle for the row cycle, and be lost completely in the column address cycle which would be wrong. I propose to modify that (unhelpful) comment.
User avatar
aneves
Experienced Member
Posts: 93
Joined: Wed Sep 16, 2015 2:38 pm

Post by aneves »

I think I was able to reproduce a similar problem in a much smaller example.

Here is the code:

Code: Select all

#include <xs1.h>
#include <platform.h>
//#include <xscope.h>
#include <stdio.h>
#include <stdlib.h>
#include "sdram.h"

#define BITS_IN_BYTE 8
#define TEMP_BUFF_SIZE 2048

#define SERVER_TILE 1
on tile[SERVER_TILE] : out buffered port:32   sdram_dq_ah = XS1_PORT_16B;
on tile[SERVER_TILE] : out buffered port:32   sdram_cas   = XS1_PORT_1D;
on tile[SERVER_TILE] : out buffered port:32   sdram_ras   = XS1_PORT_1C;
on tile[SERVER_TILE] : out buffered port:8    sdram_we    = XS1_PORT_1B;
on tile[SERVER_TILE] : out port               sdram_clk   = XS1_PORT_1A;
on tile[SERVER_TILE] : clock                  sdram_cb    = XS1_CLKBLK_1;

/*
void xscope_user_init(void){
    xscope_register(0);
    xscope_config_io(XSCOPE_IO_BASIC);
}
*/

void application(streaming chanend c_server) {

    unsigned const numBits = ( 1 << 20 ) * 256; // 256 Megabits is capapcity of sdram chip
    unsigned const numBytes = numBits / BITS_IN_BYTE ;
    unsigned const numWords = numBytes / sizeof(unsigned);
    unsigned remainingWords = numWords;

    unsigned address = 0;
    int result = 1;
    s_sdram_state sdramState;

    unsigned localBuffer[TEMP_BUFF_SIZE];
    unsigned * movable movPtrLocalBuffer = localBuffer;

    unsigned numWrites = numWords / TEMP_BUFF_SIZE;
    if (numWords % TEMP_BUFF_SIZE) numWrites++;
    unsigned wordsToWrite = TEMP_BUFF_SIZE;

    for (int i = TEMP_BUFF_SIZE; i--;) movPtrLocalBuffer[i] = 0xdeadbeef; // Initialize buffer with initial pattern

    sdram_init_state(c_server, sdramState);

    for (int i = 0; i < numWrites; i++){ //Write inital pattern 0xdeadbeef

        result = sdram_write(c_server, sdramState, address, wordsToWrite, move(movPtrLocalBuffer));
        if (result == 0) sdram_complete(c_server, sdramState, movPtrLocalBuffer);
        else{
            printf("sdram_write returned %d while writing inital pattern.\n", result);
            return;
            }

        address += wordsToWrite;
        remainingWords -= wordsToWrite;
        if (remainingWords < TEMP_BUFF_SIZE) wordsToWrite = remainingWords;


        }

    // Reset some previouslu used variables

    address = 0;
    remainingWords = numWords;
    wordsToWrite = TEMP_BUFF_SIZE;

    // Start write loop

    for (int i = 0; i < numWrites; i++){

        for (int j = 0; j < TEMP_BUFF_SIZE; j++){
            unsigned index = (TEMP_BUFF_SIZE * i) + j;
            movPtrLocalBuffer[j] = index; // Fill buffer with index/address
        }

        result = sdram_write(c_server, sdramState, address, wordsToWrite, move(movPtrLocalBuffer));
        if (result == 0) sdram_complete(c_server, sdramState, movPtrLocalBuffer);
        else{
            printf("sdram_write returned %d while i=%d.\n", result, i);
            return;
        }

        address += wordsToWrite;
        remainingWords -= wordsToWrite;
        if (remainingWords < TEMP_BUFF_SIZE) wordsToWrite = remainingWords;
    }

    // Reset the same variables again for read loop

    address = 0;
    remainingWords = numWords;
    wordsToWrite = TEMP_BUFF_SIZE;

    // Start the read loop
    for (int i = 0; i < numWrites; i++){

        result = sdram_read(c_server, sdramState, address, wordsToWrite, move(movPtrLocalBuffer));
        if(result == 0) sdram_complete(c_server, sdramState, movPtrLocalBuffer);
        else{
            printf("sdram_read returned %d while i=%d.\n", result, i);
            return;
        }

        for(int j = 0; j < TEMP_BUFF_SIZE; j++){

            unsigned predictedValue = j + (i * TEMP_BUFF_SIZE);
            if(movPtrLocalBuffer[j] != predictedValue) printf("ERROR! Read value: %d. Expected %d\n", movPtrLocalBuffer[j], predictedValue);
        }

        address += wordsToWrite;
        remainingWords -= wordsToWrite;
        if(remainingWords < TEMP_BUFF_SIZE) wordsToWrite = remainingWords;

    }

    printf("SDRAM demo completed successfully\n");
}

int main() {
  streaming chan c_sdram[1];

  par {
      on tile[SERVER_TILE]:sdram_server(c_sdram, 1,
              sdram_dq_ah,
              sdram_cas,
              sdram_ras,
              sdram_we,
              sdram_clk,
              sdram_cb,
              3, // CAS Latency
              256, // Row words
              16, // col_bits
              9, // col_address_bits
              13, // row_address_bits
              2, // bank_address_bits
              64,
              8192,
              5);

    on tile[SERVER_TILE]: application(c_sdram[0]);

  }
  return 0;
}
When attempting to write the entire capacity of the chip, I observe the following output during the last loop where the reading and comparing happens:

Code: Select all

ERROR! Read value: 2097152. Expected 0
ERROR! Read value: 2097153. Expected 1
ERROR! Read value: 2097154. Expected 2
ERROR! Read value: 2097155. Expected 3
ERROR! Read value: 2097156. Expected 4
ERROR! Read value: 2097157. Expected 5
ERROR! Read value: 2097158. Expected 6
ERROR! Read value: 2097159. Expected 7
ERROR! Read value: 2097160. Expected 8
ERROR! Read value: 2097161. Expected 9
ERROR! Read value: 2097162. Expected 10
ERROR! Read value: 2097163. Expected 11
So when reading from SDRAM, at address 0 it reads what should have been written at address 2097152. However it looks like during the earlier write loop, when it attempts to write to address 2097152, it loops back to address 0.

Any ideas?
User avatar
infiniteimprobability
XCore Legend
Posts: 1126
Joined: Thu May 27, 2010 10:08 am
Contact:

Post by infiniteimprobability »

Thanks for the reduced test case - this makes things a lot easier to reproduce which ultimately gets us to a fix quicker!
So when reading from SDRAM, at address 0 it reads what should have been written at address 2097152. However it looks like during the earlier write loop, when it attempts to write to address 2097152, it loops back to address 0.

Any ideas?
Firstly, the issue at 2,097,188 (36 bytes after the bank boundary) seems to have gone!?

From what you are seeing with this case, at long-word address 2,097,152, you are on the same row/column as long-word address 0, but on a different bank.

256Mb = 256*1024*1024 = 268,435,456b.. /32 = 8,388,608 long-words. 4 banks so /4 = 2,097,152 long-words per bank

What you are seeing (wrap around at 1/4 of the chip) would be the case if the bank signals (specifically BA0) on SDRAM were stuck.. Can you confirm the connections between BA0/1 and the xcore? It should be as follows:

Code: Select all

BA0/DQ13 <-> Port 16 bit 13
BA1/DQ14 <-> Port 16 bit 14
[/quote]
User avatar
aneves
Experienced Member
Posts: 93
Joined: Wed Sep 16, 2015 2:38 pm

Post by aneves »

[quote="infiniteimprobability"]

What you are seeing (wrap around at 1/4 of the chip) would be the case if the bank signals (specifically BA0) on SDRAM were stuck.. Can you confirm the connections between BA0/1 and the xcore? It should be as follows:

Code: Select all

BA0/DQ13 <-> Port 16 bit 13
BA1/DQ14 <-> Port 16 bit 14
[/quote]

You mean verify the connections on our hardware?

I'm confused about the pin mappings you've stated, According to the lib sdram documentation, BA0 and BA1 should be shared with DQ14 and DQ15 respectively. Is that what you're talking about or was it something else?
User avatar
infiniteimprobability
XCore Legend
Posts: 1126
Joined: Thu May 27, 2010 10:08 am
Contact:

Post by infiniteimprobability »

You mean verify the connections on our hardware?

I'm confused about the pin mappings you've stated, According to the lib sdram documentation, BA0 and BA1 should be shared with DQ14 and DQ15 respectively. Is that what you're talking about or was it something else?
Aha - I think we have a smoking gun! Your bank signals are coming out 1 bit too low for your hardware. Unfortunately, the issue is due to documentation that is inconsistent with the library code, for which I apologise. Fortunately however, all of the functionality is soft, so it's an easy fix:

You need to change server.xc to:

Code: Select all

#define BANK_SHIFT          (14)
for your hardware and the guidance in the documentation.

I will add a bug report against this library. Let us know if this fixes it.
User avatar
aneves
Experienced Member
Posts: 93
Joined: Wed Sep 16, 2015 2:38 pm

Post by aneves »

OK, we have progress!

Changing BANK_SHIFT to 14 allows my small example that I posted to complete successfully.

However, my original test bed still shows the same problem expect this time the aliasing boundary is much larger. Now if I write and read 8,388,645 bytes I get the same effect as I previously described where the first byte in the read buffer is the last byte in the write buffer.

Since the example code I posted appears to be functioning now after making the modification in server.xc, I can only assume it is a bug in my own code. I'll do some refactoring and debugging on my end and post back later with what I find.
infiniteimprobability wrote: I propose to modify that (unhelpful) comment.
Looks like it turned out to be quite helpful in my case! ;-)
User avatar
infiniteimprobability
XCore Legend
Posts: 1126
Joined: Thu May 27, 2010 10:08 am
Contact:

Post by infiniteimprobability »

OK, we have progress!
Changing BANK_SHIFT to 14 allows my small example that I posted to complete successfully.
Good! I'm glad that helped.
However, my original test bed still shows the same problem expect this time the aliasing boundary is much larger. Now if I write and read 8,388,645 bytes I get the same effect as I previously described where the first byte in the read buffer is the last byte in the write buffer.
8,388,645 - Well the device only has 8,388,608 (256*1024*1024/32) addressable long words so I would expect some sort of wrapping. However, there is still this 36 long word effect, although arguably, writing beyond the end of memory is not a good thing to do.
I'll do some refactoring and debugging on my end and post back later with what I find.
Great thanks - This 36 long word thing is strange and would be good to understand, even if you are writing over the end of the device at the time.
Post Reply