iget can block if the inode is being initialised or freed
by a different thread. It is not acceptable for the cleaner
to block in these cases as the other thread my need to trigger
a checkpoint.
So use special match/set functions to ensure we never
block, and use B_Async to check if we need to wake the
cleaner when done with an inode.
We don't really want to do anything of lafs_write_super
as we write the superblock when needed anyway.
However lafs_sync_fs needs to do what lafs_write_super was doing,
at least sometimes.
lafs_sync_fs will now force a checkpoint exactly when s_dirt is
set. So revise those setting a little - I think we only want this if
there are dirty inodes to flush. but that needs to be thought about
more when I fix write_inode.
And data blocks in realloc will have been destroyed in
erase_dblock, but there could legitimately be Realloc index blocks
still, so allow them to be handled.
Don't set I_Trunc until pages are invalidated and trunc_next is set.
The block could already be subject to orphan handling, as unlink
sets that up before truncation happens.
So make sure not to set I_Trunc until we a really ready for the
orphan-inode truncation handling to happen.
Without this truncation can race with the cleaner and weird thinks
can happen.
As we processes blocks from the segment in order, it is best
to keep them in order for later processing.
They will be sorted again when being added to a cluster,
but the more we help here, the better.
newblocks is the count of new blocks written to the filesystem in this
checkpoint (roughly the amount of work that roll-forward would have to
do). We use it to trigger new checkpoints.
So we need to reset it after each checkpoint.
Make sure the segment being written is never cleaned.
Cleaning the current segment would be a bad idea as it's
usage count isn't really representative of anything useful.
So leave it in the table flags as 'active' to avoid it
becoming cleanable, and remove it when the segment is finished
with.
More issues with wc->seg being explicitly unset at certain times.
We need to clear wc->seg at the end of a cleaning segment when we
choose not to add another, and we need to cope correctly when such
a segment is found.
When we unified the two loops in lafs_cluster_allocate,
we broke handling for a nearly-full cluster. We need to
require wc->remaining is at least 1 before we even
consider a cluster_insert.
NeilBrown [Tue, 29 Jun 2010 11:45:52 +0000 (21:45 +1000)]
Remove all to lafs_io_wake in lafs_cluster_allocate
We only need io_wake when we unlock or clear writeback, and neither of
those happen here so discard the call, put in a 'break' instead and
turn the (hard to read) do loop into a for loop.
NeilBrown [Tue, 29 Jun 2010 11:34:24 +0000 (21:34 +1000)]
combine two loops in lafs_cluster_allocate
There are two loops which try to cluster_flush to get enough space.
One calls new_segment, the other assumes cluster_flush will do
that, which it might or might not.
Combine these into a single loop, and move the handling of
error from new_segment closer to the call.
NeilBrown [Mon, 28 Jun 2010 22:04:45 +0000 (08:04 +1000)]
Handle write clusters which point to themselves.
This can happen at the end of a cleaner segment, and in
general it is best to be cautious. So if the next pointer
isn't further along in this segment, don't follow it.
NeilBrown [Mon, 28 Jun 2010 10:27:44 +0000 (20:27 +1000)]
Add EmergencyClean mode
In this mode we are nearly full.
Cleaning just goes for the segment with the most space
even if it is quite new.
Allocation failures return ENOSPC rather than EAGAIN
We clean even if it doesn't look like it will help much.
The heuristic for switching in an out is rather odd...
NeilBrown [Mon, 28 Jun 2010 09:40:32 +0000 (19:40 +1000)]
Reserve space for cleaner segments.
Now that we can reserve space specifically for cleaner
segments, do so and limit the number of cleaned segments
to the available number of cleaner segments.
NeilBrown [Mon, 28 Jun 2010 08:10:51 +0000 (18:10 +1000)]
Revise space allocation for cleaning.
We prefer to allocate whole segments for cleaning, but can only
do that if there is enough space.
If we cannot allocate whole segments, then just cleaning to the
main segment is perfectly acceptable.
So allow a 'clean_reserved' number which is a number of blocks that
have been reserved for cleaning - normally some number of segments.
The cleaner write whole segments while this number is big enough,
then gives up so the remainder will go to the main segment and not
create partial clean segments.
CleanSpace now never fails. The next patch will cause the cleaner
to be more reserved in how much it asks for.
NeilBrown [Mon, 28 Jun 2010 05:29:44 +0000 (15:29 +1000)]
Report directory size without holes.
Holes in a directory are an implementation details that does not need
to be exposed in i_size, and doing so is confusing and could leak info
about the hash used.
So when more than one block is used, report size as block size times
number of blocks.
NeilBrown [Mon, 28 Jun 2010 04:51:10 +0000 (14:51 +1000)]
Don't clear PinPending in blocks in the inode map.
We do not enforce exclusive access to these blocks, so it is not safe
to clear PinPending - some other thread might be allocating a nearby
inode and might need PinPending to remain.
It will get cleared at the next checkpoint or when the refcount on the
block hits zero, so there is no pressing need to clear it.
NeilBrown [Mon, 28 Jun 2010 03:33:05 +0000 (13:33 +1000)]
wait for pending truncate in delete_inode
If we truncate then delete, the truncate could be on-going.
Safest to wait for it to finish before deleting (and this
truncating from 0).
This makes it more consistent with lafs_truncate.
NeilBrown [Mon, 28 Jun 2010 02:33:18 +0000 (12:33 +1000)]
Fix incorrect de-ref of ->my_inode
my_inode may not be set on a block on writeout - e.g. if it was
just cleaned.
As my_inode does not disappear once set (while a ref is held
on the block) it is safe to simply test it.
NeilBrown [Mon, 28 Jun 2010 02:19:39 +0000 (12:19 +1000)]
Fix races between truncate and cleaner.
Not only do we need to recheck the size after putting
the block on the clean list, we also need to check
for inodes that have been cleared. (type == 0).
NeilBrown [Sat, 26 Jun 2010 03:26:37 +0000 (13:26 +1000)]
Pin SegmentMap block when they might need to be dirtied.
This is more in-keeping with other practices and not that SegmentMap
blocks are handles carefully by the cleaner and checkpoint, it is
safe to do this.
They stay pinned until they are no-longer referenced. This may keep
some Credits unavailable but that is not a big cost.
When we pin these we don't hold or need a phase lock, so don't
require it.
Now that we alway pin segments when first used (free_get) we don't
need to prealloc in lafs_seg_move.
NeilBrown [Sat, 26 Jun 2010 01:38:08 +0000 (11:38 +1000)]
Revise which blocks need N* credits.
I think it is just those that might be phase-flipped.
Let's hope that is right.
I think the others were there due to cleaning issues which
have not been resolved.
NeilBrown [Sat, 26 Jun 2010 01:26:46 +0000 (11:26 +1000)]
Be Careful about cleaning PinPending blocks.
PinPending blocks must never be written to the cleaner segment
as they might still get dirtied and need to be written in this phase,
but the cleaner will have taken their uninc credit.
So if we need to clean a PinPending block, just mark it dirty and
wait for it to be unpinned or written normally.
NeilBrown [Sat, 26 Jun 2010 01:16:10 +0000 (11:16 +1000)]
Change flushing of space-accounting blocks.
Space-accounting blocks need to be flushed very late in the
checkpoint.
We were special casing these, but in an awkward way.
Change it so that these blocks are pinned, but that a checkpoint
doesn't handle them straight away but rather performs a phase_flip
and then queues them for later handling.
This means that we get more consistent behviour of pinned data blocks
and writepage doesn't need to special-base the flushing of segment
usage blocks.
NeilBrown [Fri, 25 Jun 2010 10:34:52 +0000 (20:34 +1000)]
Add debug tracing to unlink.
Have had strange code of unlink failing to find the target
file. So add lots of tracing in the hope that it will happen
again.
It might be sensitive to the hash chosen.
NeilBrown [Fri, 25 Jun 2010 09:17:29 +0000 (19:17 +1000)]
Add cluster list tracking to print_tree
In print tree, where we try to print which 'lru' list a block
is on, also check the write-cluster lists, both in preparation,
and waiting for Writeout.
NeilBrown [Fri, 25 Jun 2010 09:15:32 +0000 (19:15 +1000)]
Add has_ref to help debugging.
It is sometimes helpful to BUG_ON whether a block has a certain
ref or not.
So add "has_ref" which returns -1 if we don't know (debugging
disabled) or 0/1 depending on whether ref is held.
NeilBrown [Wed, 23 Jun 2010 07:00:23 +0000 (17:00 +1000)]
Allow writers to block while the cleaner makes a little progress.
We record how much progress is required, and allow to wait
for that much progress to happen at which point a checkpoint
happens.
Also add 'free_segs' similar to 'free_blocks' (and counting blocks)
which counts the number of blocks in free segs, not including
any current segs. This forced allocators to wait sooner and
may be more appropriate.
NeilBrown [Fri, 25 Jun 2010 06:30:18 +0000 (16:30 +1000)]
Improve flushing of 'cleaner' clusters.
There is no need for the cleaner to ever wait for block which
have been written. Once the write has been requested, the block
will not be Realloc any more, and so will not get back onto the
clean_leaf list anyway.
So just flush out the cluster when everything is done.
Earlier flushes will happen when a segment gets full.
Just to be safe, also flush the cleaner cluster before a checkpoint.
If there is anything awaiting flushing, it will be invisible
to the checkpoint process, but will hold other blocks pinned
so the checkpoint will not be able to proceed.
NeilBrown [Wed, 23 Jun 2010 02:29:57 +0000 (12:29 +1000)]
Don't allow memory flush to write out segusage blocks.
We always write these after a checkpoint, and there is little to be
gained by writing them earlier, and doing so causes their
dirty status to be lost, which is bad.
They should be treated much like PinPending blocks, but they
are not PinPending as they are written later in the checkpoint.
NeilBrown [Wed, 23 Jun 2010 02:23:50 +0000 (12:23 +1000)]
Fix inode_orphan_handler issues.
1/ a stray ';' caused a while loop not to work.
2/ If the for loop finds a block with a 'primary' reference,
just incorporating it won't help. We need to find the last
block so we know it has not primary reference, so it will
get unpinned by the lafs_cluster_allocate call, and so
will remove the primary reference.
NeilBrown [Tue, 22 Jun 2010 05:32:05 +0000 (15:32 +1000)]
Honour EmptyIndex during index lookup.
If we find an EmptyIndex that isn't first in the parent, we must
choose an earlier block.
We must check EmptyIndex after getting the lock.
If we have to drop a lock to do IO, and the unlocked block has a
->parent pointer, then we need to retry from the top.
'next' needs special care as it is could point to an
EmptyIndex block, so it is possible for leafs earlier in the tree to
have higher fileaddr (unlikely but possible).
NeilBrown [Tue, 22 Jun 2010 02:35:32 +0000 (12:35 +1000)]
Add EmptyIndex flag.
This signals that an index block is known to be empty and
should normally be ignored.
It may never be set on an InoIdx block.
Normally it stays set once set. However for the first index block
in a parent, it can be cleared again if any children appear.
NeilBrown [Tue, 22 Jun 2010 02:32:07 +0000 (12:32 +1000)]
Filter empty block from uninc change before incorporation.
It is possible (though unusual) for an uninc chain to have
two block with the same fileaddr, one that is empty and being ignored,
and one that is newly split off and needs to be incorporated.
We need to detect this possibility after sorting and discard the
empty block so it doesn't confuse further incorporation.
NeilBrown [Tue, 22 Jun 2010 02:13:45 +0000 (12:13 +1000)]
Delay hashing of index blocks until they are incorporated.
We don't need an index block in the hash table until its
address is in the parent, as until then we will never try a lookup.
And it is good to delay it as it is possible for there to be two
blocks with the same address, one that is empty and thus ignored
mostly, and one that has since split of an earlier child.
While this is unlikely, we don't want that split-off block to
appear in the hash table until both have been incorporated.
NeilBrown [Mon, 21 Jun 2010 04:55:26 +0000 (14:55 +1000)]
allocate_block fixes.
Try to set up slightly new rules....
1/ Adding an address as unincorporated requires just a spinlock,
the inode private_lock
2/ The block with an address being added is not iolock, but is
Writebehind
3/ incorporation happens under iolock, removing the list of
pending addresses also happens here. so under iolock
addresses can be added to list but not removed (unless I'm doing
the removing).
projects / LaFS.git / log