Hallvard Furuseth [Mon, 5 Aug 2013 07:55:48 +0000 (09:55 +0200)]
Clarify doc: mdb_copy, nested txns, mdb_drop().
mdb_copy: Does not copy lockfile. Can trigger file growth.
mdb_txn_begin(): Clarify usage restrictions.
mdb_drop(): State what to do rather than what will be done, since
closing the handle could otherwise be read as happening even at failure.
Howard Chu [Wed, 31 Jul 2013 15:09:40 +0000 (08:09 -0700)]
Tweak prev commit again
Make sure errors are propagated from init_meta
Howard Chu [Tue, 30 Jul 2013 20:44:28 +0000 (13:44 -0700)]
Fix typo in Win32 branch
Howard Chu [Tue, 30 Jul 2013 19:47:12 +0000 (12:47 -0700)]
Cleanup prev commit
Loop on copyfd meta write, since pipes may return after partial write.
Howard Chu [Tue, 30 Jul 2013 17:22:12 +0000 (10:22 -0700)]
ITS#7652 fix I/O error checks
partial revert of
d6d2638acc245116b8f091ac425b6700d06c4713 and
26a25df5fcc2fcddae6597a61c1b867fc27c568b
The original code was already tested and working correctly.
Howard Chu [Mon, 29 Jul 2013 00:02:51 +0000 (17:02 -0700)]
Tweak mdb_envinfo numreaders
Return the actual shared reader count when it exists, not
just the current process env's reader count.
Howard Chu [Fri, 26 Jul 2013 17:19:54 +0000 (10:19 -0700)]
ITS#7615 use shorter names for semaphores
NetBSD can only handle up to 14 chars, we were using 21. Now
we encode to 15, and for NetBSD truncate the last char.
Howard Chu [Fri, 19 Jul 2013 16:57:33 +0000 (09:57 -0700)]
Tweak reader_pid check
Check again after acquiring rmutex. Avoids potential issue with
a duplicate pid coming in between initial check and rmutex.
Howard Chu [Fri, 19 Jul 2013 16:55:10 +0000 (09:55 -0700)]
Get pid lock outside of rmutex
Avoid holding rmutex for longer than necessary.
Howard Chu [Thu, 18 Jul 2013 22:24:09 +0000 (15:24 -0700)]
Tweak reader checks
Use mti_numreaders for loop limit, not me_maxreaders.
Howard Chu [Thu, 18 Jul 2013 17:40:21 +0000 (10:40 -0700)]
Add mdb_reader_check()
Howard Chu [Thu, 18 Jul 2013 16:11:09 +0000 (09:11 -0700)]
Split MDB_VERSION to MDB_DATA/MDB_LOCK VERSION
Howard Chu [Thu, 18 Jul 2013 16:00:51 +0000 (09:00 -0700)]
Tweak reader_list
Howard Chu [Thu, 18 Jul 2013 15:33:24 +0000 (08:33 -0700)]
Tweak mdb_stat(1)
Don't obtain reader txn before displaying reader table. Exit
after reader table if no other DB query options were given.
Howard Chu [Thu, 18 Jul 2013 14:41:11 +0000 (07:41 -0700)]
Add mdb_reader_list()
Dump the active slots in the reader table.
Howard Chu [Mon, 15 Jul 2013 17:57:13 +0000 (10:57 -0700)]
Add mdb_dbi_flags()
Retrieve the flags from a DB handle.
Howard Chu [Sun, 14 Jul 2013 23:53:04 +0000 (16:53 -0700)]
Fix child txn dirty_room counts in spill/unspill
Don't count pages twice if they're already accounted in an ancestor txn.
Howard Chu [Sun, 14 Jul 2013 15:28:26 +0000 (08:28 -0700)]
More for stale sub-cursor flags
Same fix for cursor_first/last.
Howard Chu [Sun, 14 Jul 2013 15:20:18 +0000 (08:20 -0700)]
Fix stale sub-cursor C_INIT flag
Whenever we enter cursor_set() the sub-cursor's flag must be
cleared. If the new cursor position has valid subdata it will
be initialized again, if not then the sub-cursor has nothing
to point to.
Howard Chu [Fri, 12 Jul 2013 20:55:18 +0000 (13:55 -0700)]
Tweak comments, defaults should be OK already
Howard Chu [Fri, 12 Jul 2013 20:36:05 +0000 (13:36 -0700)]
Bump version to 0.9.7
Hallvard Furuseth [Fri, 12 Jul 2013 09:30:33 +0000 (11:30 +0200)]
Also set/clear P_KEEP in parent txn's cursors
Howard Chu [Thu, 11 Jul 2013 20:09:47 +0000 (22:09 +0200)]
Spill pages, take 3
Howard Chu [Thu, 11 Jul 2013 20:09:47 +0000 (22:09 +0200)]
Delay touching pages until cursor is positioned.
This avoids unnecessary rewrites of pages that do not change.
(Restructuring for upcoming mdb_page_spill work.)
Hallvard Furuseth [Thu, 11 Jul 2013 20:09:46 +0000 (22:09 +0200)]
Simplify: Always set C_UNTRACK for tracked cursors.
TODO: Rename C_UNTRACK to C_TRACKED. Omitted now for readability.
The current name is because it's lazy: not always set when tracked.
Hallvard Furuseth [Thu, 11 Jul 2013 20:09:46 +0000 (22:09 +0200)]
Save freelist using proper mdb_cursor_put().
(Restructuring for upcoming mdb_page_spill work.)
mdb_freelist_save() can't just Get() the destination, since
mdb_page_spill() may have put the destination in the read-only map.
TODO: Can this new put() modify the freelist, which would break it? The
final iteration's put() can shorten the node, the rest uses MDB_CURRENT.
We could set P_KEEP on dirty freeDB leaves and ovpages, since they are
all about to be modified. But the code in this commit must stay anyway,
if mdb should support dropping a 256G DB. I.e. too big for dirty_list.
Howard Chu [Thu, 11 Jul 2013 20:09:46 +0000 (22:09 +0200)]
Move code out to mdb_page_dirty()
Howard Chu [Thu, 11 Jul 2013 20:09:46 +0000 (22:09 +0200)]
Factor out parent
Howard Chu [Fri, 12 Jul 2013 19:53:35 +0000 (12:53 -0700)]
Fix env_read_header() on Windows
Commit
d6d2638acc245116b8f091ac425b6700d06c4713 broke read
on zero-length files.
Hallvard Furuseth [Wed, 10 Jul 2013 20:11:44 +0000 (22:11 +0200)]
Do not follow uninited cursors' page pointers.
Nor uninited cursors' subcursors' page pointers.
Howard Chu [Wed, 10 Jul 2013 18:03:51 +0000 (11:03 -0700)]
Fix rebalance/cursor adjust
When collapsing root, must also move cursor index down,
not just the page pointer.
Also in mtest, break from NEXT loops on error, otherwise it just
prints the previous key/data again, which looks confusing.
Howard Chu [Wed, 10 Jul 2013 15:49:29 +0000 (08:49 -0700)]
Cursors: Clear C_EOF when clearing C_INITIALIZED
Howard Chu [Tue, 9 Jul 2013 21:21:35 +0000 (14:21 -0700)]
Fixup other cursors after delete op
Hallvard Furuseth [Sun, 7 Jul 2013 15:14:38 +0000 (17:14 +0200)]
ITS#7515 Reject conflicting page versions.
If mdb_page_touch() sees a page in txn's dirty_list, that
is the page version txn's cursors should have. Fail if
the user may be seeing and depending on another version.
Hallvard Furuseth [Sun, 7 Jul 2013 15:13:27 +0000 (17:13 +0200)]
ITS#7515 Fix tracking of parent txn's cursors.
Restore mc_flags and xcursors, they were tracked but not merged.
Simplify: Track parent txn's original cursors after backing them
up, instead of tracking copies and merging them back at commit.
Hallvard Furuseth [Sun, 7 Jul 2013 15:13:27 +0000 (17:13 +0200)]
Simplify MDB_cursor: Drop flags C_ALLOCD,C_SHADOW.
Hallvard Furuseth [Sat, 6 Jul 2013 19:42:45 +0000 (21:42 +0200)]
Silence more uninit warnings
Howard Chu [Tue, 2 Jul 2013 09:23:49 +0000 (02:23 -0700)]
Silence uninit warning in prev commit
Howard Chu [Tue, 2 Jul 2013 09:19:17 +0000 (02:19 -0700)]
Tweaks for MDB_MULTIPLE
Terminate loop on intermediate failures, return count of written items,
document usage.
Howard Chu [Mon, 1 Jul 2013 21:53:29 +0000 (14:53 -0700)]
Howard Chu [Mon, 1 Jul 2013 20:41:23 +0000 (13:41 -0700)]
ITS#7635 fix read txn potential data race
Howard Chu [Sun, 30 Jun 2013 14:40:02 +0000 (07:40 -0700)]
Fix uninit warnings, lseek usage
Hallvard Furuseth [Wed, 26 Jun 2013 16:02:52 +0000 (18:02 +0200)]
Fix alloc/free issues.
Page leak, mdb_page_alloc(). On error, don't shorten me_pghead.
Memleak, mdb_ovpage_free(). Free page or keep it in dirty_list.
Bad MIDL, mdb_midl_need(). Fix midl[-1] (allocated size).
Hallvard Furuseth [Wed, 26 Jun 2013 16:02:48 +0000 (18:02 +0200)]
Factor out some vars, simplify.
Hallvard Furuseth [Wed, 26 Jun 2013 16:02:26 +0000 (18:02 +0200)]
Makefile/user-macro comments.
Hallvard Furuseth [Wed, 26 Jun 2013 16:02:17 +0000 (18:02 +0200)]
Tweak I/O, fix last commit.
Hallvard Furuseth [Sat, 22 Jun 2013 21:15:10 +0000 (23:15 +0200)]
Improve MDB error handling, drop seek calls.
Catch I/O errors. Do nothing between OS call failure and ErrCode().
Do not use errno after non-OS-errors like write() >= 0, which could
give a failure return of success (errno 0) or some irrelevant error
code. Drop seek calls, use pwrite/pread/Windows OVERLAPPED offset.
Hallvard Furuseth [Sat, 22 Jun 2013 21:01:30 +0000 (23:01 +0200)]
Fix Windows I/O.
Don't put a 64-bit filesize in a 32-bit int before shifting
down. Always pass &sizehi to SetFilePointer->maxsize, so
sizelo not is treated a signed distance. Hide unused vars
when _WIN32. Reinitialize OVERLAPPED before reuse.
Hallvard Furuseth [Sat, 22 Jun 2013 20:17:41 +0000 (22:17 +0200)]
Catch more MDB errors. DPRINTF in mdb_env_reset0.
Hallvard Furuseth [Sat, 22 Jun 2013 20:10:43 +0000 (22:10 +0200)]
Tweak MIDLs, catch errors.
Grow midls earlier in order to catch errors earlier. Use
mdb_midl_need() instead of mdb_midl_grow(), then mdb_midl_xappend()
needs no error checks. Factor out mdb_midl_append_range().
Hallvard Furuseth [Sat, 22 Jun 2013 10:30:04 +0000 (12:30 +0200)]
Factor out MDB variables/expressions, cleanup.
mdb_page_malloc(): Take a txn arg instead of a cursor.
Hallvard Furuseth [Sat, 22 Jun 2013 09:56:04 +0000 (11:56 +0200)]
Rearrange MDB dirty page code.
Split out mdb_dpage_free(), mdb_page_flush() and clean up.
Hallvard Furuseth [Thu, 20 Jun 2013 05:41:35 +0000 (07:41 +0200)]
Simplify mdb_page_alloc().
Merge if() branches. Restore retry=500 when MDB_PARANOID, for clarity.
Hallvard Furuseth [Thu, 20 Jun 2013 05:41:35 +0000 (07:41 +0200)]
ITS#7620: Keep empty IDLs. Tweak mdb_page_alloc().
MDB_env.me_pghead: Don't free it when empty. mdb_ovpage_free()
needs it, but cannot allocate it.
mdb_midl_alloc(): Fill in length=0.
mdb_page_alloc(): Also Skip freeDB if txnid<3, instead of <4,
and consistently DPRINTF consumed IDLs.
Howard Chu [Mon, 17 Jun 2013 20:26:11 +0000 (22:26 +0200)]
ITS#7623 Clear P_SUBP on conversion from fake page
Hallvard Furuseth [Thu, 13 Jun 2013 06:58:25 +0000 (08:58 +0200)]
ITS#7515 Nested MDB txns: Inherit txn flags.
Committing a nested txn lost the MDB_TXN_DIRTY flag
in the parent, unless the child had set it too.
Hallvard Furuseth [Thu, 13 Jun 2013 06:58:24 +0000 (08:58 +0200)]
Clean up mdb_page_touch(), mdb_page_copy().
When copying, round up/down to aligned sizes. Skip the unused portion,
this was not done when touching a page dirty in the parent txn.
No other change in behavior.
Simplify mdb_page_touch(), including: Drop test m3==mc, the condition
is caught below. Don't "modify" the parent's pgno into the same pgno,
when a nested txn copies a parent's page into its freelist.
Hallvard Furuseth [Thu, 13 Jun 2013 06:58:24 +0000 (08:58 +0200)]
ITS#7594 Fix MDB cursor tracking with subDBs.
The tracking code should not change the current cursor.
It did when that was a C_SUB cursor, which should not be
checked against the tracked cursors but their xcursors.
However, do not bother to skip the tracking code for the
current cursor when it would not change that cursor anyway.
Hallvard Furuseth [Thu, 13 Jun 2013 06:58:24 +0000 (08:58 +0200)]
ITS#7594 Invalidate a dropped MDB DB's cursors.
Hallvard Furuseth [Thu, 13 Jun 2013 06:58:24 +0000 (08:58 +0200)]
Don't #define _GNU_SOURCE if already defined.
Hallvard Furuseth [Thu, 13 Jun 2013 06:25:25 +0000 (08:25 +0200)]
More for ITS#7620 Fix mdb_ovpage_free().
Do not binary-search dirty_list, it is unsorted when MDB_WRITEMAP.
Catch errors. In nested txns, put the page in mt_free_pgs after
all since pages dirty in a parent txn would add complexities.
Howard Chu [Wed, 12 Jun 2013 15:41:32 +0000 (08:41 -0700)]
Partial revert
c2cac4588a40480c020d320b544bc5f8e72adb11
MDB_NEXT was fine before, duh.
Hallvard Furuseth [Wed, 12 Jun 2013 15:20:42 +0000 (17:20 +0200)]
Drop me_pgfree, add mdb_freelist_save().
Split up saving me_pghead, to make me_pgfree unneeded. Also mf_pghead
is now a midl. Needed after
e7f6767ea815fe0ada1f95037dfdec176ec4d5bb
("Return fresh overflow pages to current pghead").
Tweak MDB_DEBUG freelist output, make it ascending.
Howard Chu [Wed, 12 Jun 2013 00:13:08 +0000 (17:13 -0700)]
Fix CURSOR_NEXT/PREV on emptied DB
Howard Chu [Sat, 8 Jun 2013 21:10:08 +0000 (14:10 -0700)]
Make sure mdb_stat() gets valid data
Howard Chu [Sun, 5 May 2013 08:28:12 +0000 (01:28 -0700)]
Return fresh overflow pages to current pghead
And remove them from the current dirty list.
Howard Chu [Wed, 5 Jun 2013 23:13:43 +0000 (16:13 -0700)]
ITS#7594 more for subDB cursor fix
Howard Chu [Wed, 5 Jun 2013 22:23:54 +0000 (15:23 -0700)]
ITS#7594 better fix
Update the subDB cursor, don't invalidate it
Howard Chu [Thu, 30 May 2013 22:56:30 +0000 (15:56 -0700)]
tweak mdb_copy, trap signals
Howard Chu [Thu, 30 May 2013 22:33:59 +0000 (15:33 -0700)]
Windows portability fixes for prev commit
Howard Chu [Thu, 30 May 2013 20:13:33 +0000 (13:13 -0700)]
Add warning about interrupting copy
Howard Chu [Thu, 30 May 2013 20:09:28 +0000 (13:09 -0700)]
Fix prev commit
Howard Chu [Thu, 30 May 2013 20:06:12 +0000 (13:06 -0700)]
Add mdb_env_copyfd()
Allow writing backup to an already opened file handle, for piping
to tar/gzip/ssh/whatever.
Howard Chu [Sat, 25 May 2013 17:16:55 +0000 (10:16 -0700)]
Add _M_IX86 macro for MSVC
Howard Chu [Thu, 23 May 2013 15:13:08 +0000 (08:13 -0700)]
ITS#7594 De-init other subcursors in page_touch
Hallvard Furuseth [Tue, 21 May 2013 21:58:57 +0000 (23:58 +0200)]
Drop unused liblmdb MIDL-range support.
Hallvard Furuseth [Tue, 21 May 2013 21:55:13 +0000 (23:55 +0200)]
Factor out mdb_find_oldest,mdb_dlist_free,dirty_list.
Do not rescan reader table (mdb_find_oldest) after "goto again".
Skip clearing dirty_list[nonzero].mid in mdb_dlist_free(); it
was not done in mdb_reset0() anyway.
Hallvard Furuseth [Tue, 21 May 2013 21:48:27 +0000 (23:48 +0200)]
mdb_stat cleanup.
Exit with success when there was no failure.
Do not use data containing NUL as a DB name (which is a C string).
Hallvard Furuseth [Tue, 21 May 2013 20:44:51 +0000 (22:44 +0200)]
ITS#7598 Tweak MDB_<NEXT/PREV>_NODUP,fix mdb_stat.
MDB_NEXT_NODUP, MDB_PREV_NODUP: Allow for non-MDB_DUPSORT databases.
No mdb.c code changes needed.
mdb_stat.c: Use MDB_NEXT_NODUP, to avoid a crash with a DUPSORT mainDB.
Hallvard Furuseth [Tue, 21 May 2013 17:04:52 +0000 (19:04 +0200)]
ITS#7598 mdb_dbi_open(named DB): Check mainDB flags.
Reject attempts to open named databases if the main
database has flag MDB_DUPSORT or MDB_INTEGERKEY.
DUPSORT would require an xcursor for the DB, INTEGERKEY
would expect the DB name to be a binary integer.
Hallvard Furuseth [Mon, 20 May 2013 21:12:52 +0000 (23:12 +0200)]
ITS#7515 Fix nested txn touch of subpage/ovpage.
mdb_page_touch(): Don't touch a subpage, replacing with non-subpage.
mdb_cursor_put(): Don't overwrite ancestor txn's dirty overflow page.
Hallvard Furuseth [Mon, 20 May 2013 21:12:26 +0000 (23:12 +0200)]
Tweak mdb_page_malloc(),mdb_page_get() semantics.
mdb_page_malloc(): Add "number of pages" parameter.
mdb_page_get(): Add output param for how page was found.
Do not set return params on error.
mdb_cursor_put(): Catch mdb_page_get() error.
Prepares for next commit, no change in caller behavior other
than on mdb_page_get error.
Hallvard Furuseth [Sun, 19 May 2013 17:08:07 +0000 (19:08 +0200)]
More ITS#7589 followup: OVPAGES() -> mp_pages.
Hallvard Furuseth [Sun, 19 May 2013 16:38:13 +0000 (18:38 +0200)]
More ITS#7589: Fix prev commit.
mp_pages gives #pages in the ovpage. OVPAGES() no longer does.
Hallvard Furuseth [Fri, 17 May 2013 17:31:17 +0000 (19:31 +0200)]
ITS#7589 mdb_cursor_put(): Update ovpage nodesize.
Update the nodesize when overwriting an overflow page.
As before, do not attempt to shrink the page.
Howard Chu [Wed, 15 May 2013 18:50:40 +0000 (11:50 -0700)]
Include <netinet/in.h> for Solaris 8
Needed for <resolv.h>, which is needed for BYTE_ORDER.
Howard Chu [Tue, 14 May 2013 23:57:50 +0000 (16:57 -0700)]
Fix mdb_page_split - nested split
If updating a page's separator triggers a split in its parent,
we may have missed adjusting the parent's cursor position.
Howard Chu [Mon, 13 May 2013 19:36:47 +0000 (12:36 -0700)]
Doxygen cleanup
Howard Chu [Fri, 10 May 2013 12:56:16 +0000 (05:56 -0700)]
Howard Chu [Fri, 10 May 2013 00:27:29 +0000 (17:27 -0700)]
Refactor mdb_midl_append, add mdb_midl_grow()
Hallvard Furuseth [Sun, 5 May 2013 13:13:31 +0000 (15:13 +0200)]
ITS#7515 Fix nested transaction error handling.
mdb_txn_begin(): Do not free(mt_free_pgs), it needs mdb_midl_free().
mdb_txn_commit(): Catch commit(child) error.
Hallvard Furuseth [Sun, 5 May 2013 12:26:05 +0000 (14:26 +0200)]
Fix mdb_env_close(unopened MDB_env).
Do not try to scan me_dbxs in a closed/never-opened MDB_env.
Broken by
7d643d3acb94a6d7306330b550fce0e5755444a9
and
151c416b460b02cd96502a47189cda19ab651ada.
Hallvard Furuseth [Sat, 4 May 2013 10:53:43 +0000 (12:53 +0200)]
Simplify/cleanup mdb_cursor_init, C_ALLOCD.
No real change.
mdb_cursor_init() checks if it needs mx, so pass it unconditionally.
Set C_ALLOCD for shadow cursors, for clarity. (It was always set as
it should anyway from the origin cursor, which would have C_ALLOCD.)
Hallvard Furuseth [Sat, 4 May 2013 10:41:28 +0000 (12:41 +0200)]
MDB warning cleanup.
Unused function when MDB_DEBUG. Unused 'excl' param.
Hallvard Furuseth [Sat, 4 May 2013 10:34:49 +0000 (12:34 +0200)]
mdb_dcmp(): Assume the database has MDB_DUPSORT.
There was little point in returning EINVAL when not: Comparing (A,B)
and (B,A) would claim (A > B && B > A), which could confuse callers.
Hallvard Furuseth [Sat, 4 May 2013 10:34:20 +0000 (12:34 +0200)]
Update MDB doc: Cursors, DB handles, data lifetime
Hallvard Furuseth [Sat, 4 May 2013 10:34:02 +0000 (12:34 +0200)]
Allow mdb_cursor_close() after readonly txn ends.
Catch mdb_cursor_renew(write txn's cursor). Add flag C_UNTRACK, so
mdb_cursor_close need not peek inside a possibly-freed readonly txn.
Hallvard Furuseth [Sat, 4 May 2013 10:33:06 +0000 (12:33 +0200)]
Close cursors when commit(writer). Factor out code.
Close remaining cursors when committing a write txn. The doc says this
happens, and it avoids cursor tracking when updating mainDB + freeDB.
Rename mdb_cursor_merge() -> mdb_cursors_close() for code reuse,
and add a merge option. Simplify its loop a bit.
Factor out cleanup of DBIs.
Hallvard Furuseth [Sat, 4 May 2013 10:31:31 +0000 (12:31 +0200)]
Plug txn leak and MDB_NOTLS slot leaks on error.
On mdb_env_copy() error: Abort the txn.
On mdb_txn_renew0() error: Release new MDB_NOTLS reader slot.
Hallvard Furuseth [Sat, 4 May 2013 08:03:19 +0000 (10:03 +0200)]
Cleaner "Support mdb_txn_abort(a reset txn)".
Redo
8a562f560edcc6417672df453c4e626dfb4c4024 so !mt_dbxs indicates
"txn was reset", so mt_numdbs gets one magic value instead of two.