.. _fifo-dma-pusher: ============================== DMA submission to FIFOs on NV4 ============================== .. contents:: Introduction ============ There are two modes of DMA command submission: The NV4-style DMA mode and IB mode. Both of them are based on a conception of "pushbuffer": an area of memory that user fills with commands and tells PFIFO to process. The pushbuffers are then assembled into a "command stream" consisting of 32-bit words that make up "commands". In NV4-style DMA mode, the pushbuffer is always read linearly and converted directly to command stream, except when the "jump", "return", or "call" commands are encountered. In IB mode, the jump/call/return commands are disabled, and command stream is instead created with use of an "IB buffer". The IB buffer is a circular buffer of (base,length) pairs describing areas of pushbuffer that will be stitched together to create the command stream. NV4- style mode is available on NV4:GF100, IB mode is available on G80+. .. todo:: check for NV4-style mode on GF100 In both cases, the command stream is then broken down to commands, which get executed. For most commands, the execution consists of storing methods into CACHE for execution by the puller. Pusher state ============ The following data makes up the DMA pusher state: ====== ================ ======== =========================================== type name cards description ====== ================ ======== =========================================== dmaobj dma_pushbuffer :GF100 [#S]_ the pushbuffer and IB DMA object b32 dma_limit :GF100 [#S]_ [#O]_ pushbuffer size limit b32 dma_put all pushbuffer current end address b32 dma_get all pushbuffer current read address b11/12 dma_state.mthd all Current method b3 dma_state.subc all Current subchannel b24 dma_state.mcnt all Current method count b32 dcount_shadow NV5: number of already-processed methods in cmd bool dma_state.ni NV10+ Current command's NI flag bool dma_state.lenp G80+ [#I]_ Large NI command length pending b32 ref NV10+ reference counter [shared with puller] bool subr_active NV1A+ [#O]_ Subroutine active b32 subr_return NV1A+ [#O]_ subroutine return address bool big_endian NV11:G80 [#S]_ pushbuffer endian switch bool sli_enable G80+ [#S]_ SLI cond command enabled b12 sli_mask G80+ [#S]_ SLI cond mask bool sli_active NV40+ SLI cond currently active bool ib_enable G80+ [#S]_ IB mode enabled bool nonmain G80+ [#I]_ non-main pushbuffer active b8 dma_put_high G80+ extra 8 bits for dma_put b8 dma_put_high_rs G80+ dma_put_high read shadow b8 dma_put_high_ws G80+ [#O]_ dma_put_high write shadow b8 dma_get_high G80+ extra 8 bits for dma_get b8 dma_get_high_rs G80+ dma_get_high read shadow b32 ib_put G80+ [#I]_ IB current end position b32 ib_get G80+ [#I]_ IB current read position b40 ib_address G80+ [#S]_ [#I]_ IB address b8 ib_order G80+ [#S]_ [#I]_ IB size b32 dma_mget G80+ [#I]_ main pushbuffer last read address b8 dma_mget_high G80+ [#I]_ extra 8 bits for dma_mget bool dma_mget_val G80+ [#I]_ dma_mget valid flag b8 dma_mget_high_rs G80+ [#I]_ dma_mget_high read shadow bool dma_mget_val_rs G80+ [#I]_ dma_mget_val read shadow ====== ================ ======== =========================================== .. [#S] means that this part of state can only be modified by kernel intervention and is normally set just once, on channel setup. .. [#O] means that state only applies to NV4-style mode, .. [#I] means that state only applies to IB mode. Errors ====== On pre-GF100, whenever the DMA pusher encounters problems, it'll raise a DMA_PUSHER error. There are 6 types of DMA_PUSHER errors: == ================= ============================================ id name reason == ================= ============================================ 1 CALL_SUBR_ACTIVE call command while subroutine active 2 INVALID_MTHD attempt to submit a nonexistent special method 3 RET_SUBR_INACTIVE return command while subroutine inactive 4 INVALID_CMD invalid command 5 IB_EMPTY attempt to submit zero-length IB entry 6 MEM_FAULT failure to read from pushbuffer or IB == ================= ============================================ Apart from pusher state, the following values are available on NV5+ to aid troubleshooting: - dma_get_jmp_shadow: value of dma_get before the last jump - rsvd_shadow: the first word of last-read command - data_shadow: the last-read data word .. todo:: verify those .. todo:: determine what happens on GF100 on all imaginable error conditions .. _fifo-user-mmio-dma: Channel control area ==================== The channel control area is used to tell card about submitted pushbuffers. The area is at least 0x1000 bytes long, though it can be longer depending on the card generation. Everything in the area should be accessed as 32-bit integers, like almost all of the MMIO space. The following addresses are usable: ==== === ============= ================================================= addr R/W name description ==== === ============= ================================================= 0x40 R/W DMA_PUT dma_put, only writable when not in IB mode 0x44 R DMA_GET dma_get 0x48 R REF ref 0x4c R/W DMA_PUT_HIGH dma_put_high_rs/ws, only writable when not in IB 0x50 R/W ??? GF100+ only 0x54 R DMA_CGET [#O]_ nv40+ only, connected to subr_return when subroutine active, dma_get when inactive. 0x58 R DMA_MGET dma_mget 0x5c R DMA_MGET_HIGH dma_mget_high_rs, dma_mget_val_rs 0x60 R DMA_GET_HIGH dma_get_high_rs 0x88 R IB_GET [#I]_ ib_get 0x8c R/W IB_PUT [#I]_ ib_put ==== === ============= ================================================= The channel control area is accessed in 32-bit chunks, but on G80+, DMA_GET, DMA_PUT and DMA_MGET are effectively 40-bit quantities. To prevent races, the high parts of them have read and write shadows. When you read the address corresponding to the low part, the whole value is atomically read. The low part is returned as the result of the read, while the high part is copied to the corresponding read shadow where it can be read through a second access to the other address. DMA_PUT also has a write shadow of the high part - when the low part address is written, it's assembled together with the write shadow and atomically written. To summarise, when you want to read full DMA_PUT/GET/MGET, first read the low part, then the high part. Due to the shadows, the value thus read will be correct. To write the full value of DMA_PUT, first write the high part, then the low part. Note, however, that two different threads reading these values simultanously can interfere with each other. For this reason, the channel control area shouldn't ever be accessed by more than one thread at once, even for reading. On NV4:NV40 cards, the channel control area is in BAR0 at address 0x800000 + 0x10000 * channel ID. On NV40, there are two BAR0 regions with channel control areas: the old-style is in BAR0 at 0x800000 + 0x10000 * channel ID, supports channels 0-0x1f, can do both PIO and DMA submission, but does not have DMA_CGET when used in DMA mode. The new-style area is in BAR0 at 0xc0000 + 0x1000 * channel ID, supports only DMA mode, supports all channels, and has DMA_CGET. On G80 cards, channel 0 supports PIO mode and has channel control area at 0x800000, while channels 1-126 support DMA mode and have channel control areas at 0xc00000 + 0x2000 * channel ID. On GF100, the channel control areas are accessed through selectable addresses in BAR1 and are backed by VRAM or host memory - see :ref:`GF100+ PFIFO ` for more details. .. todo:: check channel numbers NV4-style mode ============== In NV4-style mode, whenever dma_get != dma_put, the card read a 32-bit word from the pushbuffer at the address specified by dma_get, increments dma_get by 4, and treats the word as the next word in the command stream. dma_get can also move through the control flow commands: jump [sets dma_get to param], call [copies dma_get to subr_return, sets subr_active and sets dma_get to param], and return [unsets subr_active, copies subr_return to dma_get]. The calls and returns are only available on NV1A+ cards. The pushbuffer is accessed through the dma_pushbuffer DMA object. On NV4, the DMA object has to be located in PCI or AGP memory. On NV5+, any DMA object is valid. At all times, dma_get has to be <= dma_limit. Going past the limit or getting a VM fault when attempting to read from pushbuffer results in raising DMA_PUSHER error of type MEM_FAULT. On pre-NV1A cards, the word read from pushbuffer is always treated as little-endian. On NV1A:G80 cards, the endianness is determined by the big_endian flag. On G80+, the PFIFO endianness is a global switch. .. todo:: What about GF100? Note that pushbuffer addresses over 0xffffffff shouldn't be used in NV4-style mode, even on G80 - they cannot be expressed in jump commands, dma_limit, nor subr_return. Why dma_put writing supports it is a mystery. The usual way to use NV4-style mode is: 1. Allocate a big circular buffer 2. [NV1A+] if you intend to use subroutines, allocate space for them and write them out 3. Point dma_pushbuffer to the buffer, set dma_get and dma_put to its start 4. To submit commands: 1. If there's not enough space in the pushbuffer between dma_put and end to fit the command + a jump command, submit a jump-to-beginning command first and set DMA_PUT to buffer start. 2. Read DMA_GET/DMA_CGET until you get a value that's out of the range you're going to write. If on pre-NV40 and using subroutines, discard DMA_GET reads that are outside of the main buffer. 3. Write out the commands at current DMA_PUT address. 4. Set DMA_PUT to point right after the last word of commands you wrote. IB mode ======= NV4-style mode, while fairly flexible, can only jump between parts of pushbuffer between commands. IB mode decouples flow control from the command structure by using a second "master" buffer, called the IB buffer. The IB buffer is a circular buffer of 8-byte structures called IB entries. The IB buffer is, like the pushbuffer, accessed through dma_pushbuffer DMA object. The address of the IB buffer, along with its size, is normally specified on channel creation. The size has to be a power of two and can be in range ???. .. todo:: check the ib size range There are two indices into the IB buffer: ib_get and ib_put. They're both in range of 0..2^ib_order-1. Whenever no pushbuffer is being processed [dma_put =dma_get], and there are unread entries in the IB buffer [ib_put!=ib_get], the card will read an entry from IB buffer entry #ib_get and increment ib_get by 1. When ib_get would reach 2^ib_order, it insteads wraps around to 0. Failure to read IB entry due to VM fault will, like pushbuffer read fault, cause DMA_PUSHER error of type MEM_FAULT. The IB entry is made of two 32-bit words in PFIFO endianness. Their format is: Word 0: - bits 0-1: unused, should be 0 - bits 2-31: ADDRESS_LOW, bits 2-31 of pushbuffer start address Word 1: - bits 0-7: ADDRESS_HIGH, bits 32-39 of pushbuffer start address - bit 8: ??? - bit 9: NOT_MAIN, "not main pushbuffer" flag - bits 10-30: SIZE, pushbuffer size in 32-bit words - bit 31: NO_PREFETCH (probably; use for pushbuffer data generated by the GPU) .. todo:: figure out bit 8 some day When an IB entry is read, the pushbuffer is prepared for reading:: dma_get[2:39] = ADDRESS dma_put = dma_get + SIZE * 4 nonmain = NOT_MAIN if (!nonmain) dma_mget = dma_get Subsequently, just like in NV4-style mode, words from dma_get are read until it reaches dma_put. When that happens, processing can move on to the next IB entry [or pause until user sends more commands]. If the nonmain flag is not set, dma_get is copied to dma_mget whenever it's advanced, and dma_mget_val flag is set to 1. dma_limit is ignored in IB mode. An attempt to submit IB entry with length zero will raise DMA_PUSHER error of type IB_EMPTY. The nonmain flag is meant to help with a common case where pushbuffers sent through IB can come from two sources: a "main" big circular buffer filled with immediately generated commands, and "external" buffers containing helper data filled and managed through other means. DMA_MGET will then contain the address of the current position in the "main" buffer without being affected by IB entries pulling data from other pushbuffers. It's thus similiar to DMA_CGET's role in NV4-style mode. The commands - pre-GF100 format =============================== The command stream, as assembled by NV4-style or IB mode pushbuffer read, is then split into individual commands. The command type is determined by its first word. The word has to match one of the following forms: ================================ ==================================== 000CCCCCCCCCCC00SSSMMMMMMMMMMM00 increasing methods [NV4+] 0000000000000001MMMMMMMMMMMMXX00 SLI conditional [NV40+, if enabled] 00000000000000100000000000000000 return [NV1A+, NV4-style only] 0000000000000011SSSMMMMMMMMMMM00 long non-increasing methods [IB only] 001JJJJJJJJJJJJJJJJJJJJJJJJJJJ00 old jump [NV4+, NV4-style only] 010CCCCCCCCCCC00SSSMMMMMMMMMMM00 non-increasing methods [NV10+] JJJJJJJJJJJJJJJJJJJJJJJJJJJJJJ01 jump [NV1A+, NV4-style only] JJJJJJJJJJJJJJJJJJJJJJJJJJJJJJ10 call [NV1A+, NV4-style only] ================================ ==================================== .. todo:: do an exhaustive scan of commands If none of the forms matches, or if the one that matches cannot be used in current mode, the INVALID_CMD DMA_PUSHER error is raised. The commands ============ There are two command formats the DMA pusher can use: NV4 format and GF100 format. All cards support the NV4 format, while only GF100+ cards support the GF100 format. NV4 method submission commands ------------------------------ ================================ ==================================== 000CCCCCCCCCCC00SSSMMMMMMMMMMM00 increasing methods [NV4+] 010CCCCCCCCCCC00SSSMMMMMMMMMMM00 non-increasing methods [NV10+] 0000000000000011SSSMMMMMMMMMMM00 long non-increasing methods [IB only] ================================ ==================================== These three commands are used to submit methods. the MM..M field selects the first method that will be submitted. The SSS field selects the subchannel. The CC..C field is mthd_count and says how many words will be submitted. With the "long non-increasing methods" command, the method count is instead contained in low 24 bits of the next word in the pushbuffer. The subsequent mthd_count words after the first word [or second word in case of the long command] are the method parameters to be submitted. If command type is increasing methods, the method number increases by 4 [ie. by 1 method] for each submitted word. If type is non-increasing, all words are submitted to the same method. If sli_enable is set and sli_active is not set, the methods thus assembled will be discarded. Otherwise, they'll be appended to the CACHE. .. todo:: didn't mthd 0 work even if sli_active=0? The pusher watches the submitted methods: it only passes methods 0x100+ and methods in 0..0xfc range that the puller recognises. An attempt to submit invalid method in 0..0xfc range will cause a DMA_PUSHER error of type INVALID_MTHD. .. todo:: check pusher reaction on ACQUIRE submission: pause? NV4 control flow commands ------------------------- ================================ ==================================== 001JJJJJJJJJJJJJJJJJJJJJJJJJJJ00 old jump [NV4+] JJJJJJJJJJJJJJJJJJJJJJJJJJJJJJ01 jump [NV1A+] JJJJJJJJJJJJJJJJJJJJJJJJJJJJJJ10 call [NV1A+] 00000000000000100000000000000000 return [NV1A+] ================================ ==================================== For jumps and calls, J..JJ is bits 2-28 or 2-31 of the target address. The remaining bits of target are forced to 0. The jump commands simply set dma_get to the target - the next command will be read from there. There are two commands, since NV4 originally supported only 29-bit addresses, and used high bits as command type. NV1A introduced the new jump command that instead uses low bits as type, and allows access to full 32 bits of address range. The call command copies dma_get to subr_return, sets subr_active to 1, and sets dma_get to the target. If subr_active is already set before the call, the DMA_PUSHER error of type CALL_SUBR_ACTIVE is raised. The return command copies subr_return to dma_get and clears subr_active. If subr_active isn't set, it instead raises DMA_PUSHER error of type RET_SUBR_INACTIVE. NV4 SLI conditional command --------------------------- ================================ ==================================== 0000000000000001MMMMMMMMMMMMXX00 SLI conditional [NV40+] ================================ ==================================== NV40 introduced SLI functionality. One of the associated features is the SLI conditional command. In SLI mode, sister channels are commonly created on all cards in SLI set using a common pushbuffer. Since most of the commands set in SLI will be identical for all cards, this saves resources. However, some of the commands have to be sent only to a single card, or to a subgroup of cards. The SLI conditional can be used for that purpose. The sli_active flag determines if methods should be accepted at the moment: when it's set, methods will be accepted. Otherwise, they'll be ignored. SLI conditional command takes the encoded mask, MM..M, ands it with the per-card value of sli_mask, and sets sli_active flag to 1 if result if non-0, to 0 otherwise. The sli_enable flag determines if the command is available. If it's not set, the command effectively doesn't exist. Note that sli_enable and sli_mask exist on both NV40:G80 and G80+, but on NV40:G80 they have to be set uniformly for all channels on the card, while G80+ allows independent settings for each channel. The XX bits in the command are ignored. GF100 commands ============== GF100 format follows the same idea, but uses all-new command encoding. ================================ ==================================== 000CCCCCCCCCCC00SSSMMMMMMMMMMMXX increasing methods [old] 000XXXXXXXXXXX01MMMMMMMMMMMMXXXX SLI conditional 000XXXXXXXXXXX10MMMMMMMMMMMMXXXX SLI user mask store [new] 000XXXXXXXXXXX11XXXXXXXXXXXXXXXX SLI conditional from user mask [new] 001CCCCCCCCCCCCCSSSXMMMMMMMMMMMM increasing methods [new] 010CCCCCCCCCCC00SSSMMMMMMMMMMMXX non-increasing methods [old] 011CCCCCCCCCCCCCSSSXMMMMMMMMMMMM non-increasing methods [new] 100VVVVVVVVVVVVVSSSXMMMMMMMMMMMM inline method [new] 101CCCCCCCCCCCCCSSSXMMMMMMMMMMMM increase-once methods [new] 110XXXXXXXXXXXXXXXXXXXXXXXXXXXXX ??? [XXX] [new] ================================ ==================================== .. todo:: check bitfield bounduaries .. todo:: check the extra SLI bits .. todo:: look for other forms Increasing and non-increasing methods work like on older cards. Increase-once methods is a new command that works like the other methods commands, but sends the first data word to method M, second and all subsequent data words to method M+4 [ie. the next method]. Inline method command is a single-word command that submits a single method with a short [12-bit] parameter encoded in VV..V field. GF100 also did away with the INVALID_MTHD error - invalid low methods are pushed into CACHE as usual, puller will complain about them instead when it tries to execute them. The pusher pseudocode - pre-GF100 ================================= :: while(1) { if (dma_get != dma_put) { /* pushbuffer non-empty, read a word. */ b32 word; try { if (!ib_enable && dma_get >= dma_limit) throw DMA_PUSHER(MEM_FAULT); if (gpu < NV1A) word = READ_DMAOBJ_32(dma_pushbuffer, dma_get, LE); else if (gpu < G80) word = READ_DMAOBJ_32(dma_pushbuffer, dma_get, big_endian?BE:LE); else word = READ_DMAOBJ_32(dma_pushbuffer, dma_get, pfifo_endian); dma_get += 4; if (!nonmain) dma_mget = dma_get; } catch (VM_FAULT) { throw DMA_PUSHER(MEM_FAULT); } /* now, see if we're in the middle of a command */ if (dma_state.lenp) { /* second word of long non-inc methods command - method count */ dma_state.lenp = 0; dma_state.mcnt = word & 0xffffff; } else if (dma_state.mcnt) { /* data word of methods command */ data_shadow = word; if (!PULLER_KNOWS_MTHD(dma_state.mthd)) throw DMA_PUSHER(INVALID_MTHD); if (!sli_enable || sli_active) { CACHE_PUSH(dma_state.subc, dma_state.mthd, word, dma_state.ni); } if (!dma_state.ni) dma_state.mthd++; dma_state.mcnt--; dcount_shadow++; } else { /* no command active - this is the first word of a new one */ rsvd_shadow = word; /* match all forms */ if ((word & 0xe0000003) == 0x20000000 && !ib_enable) { /* old jump */ dma_get_jmp_shadow = dma_get; dma_get = word & 0x1fffffff; } else if ((word & 3) == 1 && !ib_enable && gpu >= NV1A) { /* jump */ dma_get_jmp_shadow = dma_get; dma_get = word & 0xfffffffc; } else if ((word & 3) == 2 && !ib_enable && gpu >= NV1A) { /* call */ if (subr_active) throw DMA_PUSHER(CALL_SUBR_ACTIVE); subr_return = dma_get; subr_active = 1; dma_get = word & 0xfffffffc; } else if (word == 0x00020000 && !ib_enable && gpu >= NV1A) { /* return */ if (!subr_active) throw DMA_PUSHER(RET_SUBR_INACTIVE); dma_get = subr_return; subr_active = 0; } else if ((word & 0xe0030003) == 0) { /* increasing methods */ dma_state.mthd = (word >> 2) & 0x7ff; dma_state.subc = (word >> 13) & 7; dma_state.mcnt = (word >> 18) & 0x7ff; dma_state.ni = 0; dcount_shadow = 0; } else if ((word & 0xe0030003) == 0x40000000 && gpu >= NV10) { /* non-increasing methods */ dma_state.mthd = (word >> 2) & 0x7ff; dma_state.subc = (word >> 13) & 7; dma_state.mcnt = (word >> 18) & 0x7ff; dma_state.ni = 1; dcount_shadow = 0; } else if ((word & 0xffff0003) == 0x00030000 && ib_enable) { /* long non-increasing methods */ dma_state.mthd = (word >> 2) & 0x7ff; dma_state.subc = (word >> 13) & 7; dma_state.lenp = 1; dma_state.ni = 1; dcount_shadow = 0; } else if ((word & 0xffff0003) == 0x00010000 && sli_enable) { if (sli_mask & ((word >> 4) & 0xfff)) sli_active = 1; else sli_active = 0; } else { throw DMA_PUSHER(INVALID_CMD); } } } else if (ib_enable && ib_get != ib_put) { /* current pushbuffer empty, but we have more IB entries to read */ b64 entry; try { entry_low = READ_DMAOBJ_32(dma_pushbuffer, ib_address + ib_get * 8, pfifo_endian); entry_high = READ_DMAOBJ_32(dma_pushbuffer, ib_address + ib_get * 8 + 4, pfifo_endian); entry = entry_high << 32 | entry_low; ib_get++; if (ib_get == (1 << ib_order)) ib_get = 0; } catch (VM_FAULT) { throw DMA_PUSHER(MEM_FAULT); } len = entry >> 42 & 0x3fffff; if (!len) throw DMA_PUSHER(IB_EMPTY); dma_get = entry & 0xfffffffffc; dma_put = dma_get + len * 4; if (entry & 1 << 41) nonmain = 1; else nonmain = 0; } /* otherwise, pushbuffer empty and IB empty or nonexistent - nothing to do. */ }