Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Eliminate some scans + store jobs in BTree #738

Merged
merged 28 commits into from
May 30, 2023
Merged
Changes from 1 commit
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
c21b262
avoid some scans
bcantrill May 13, 2023
1e21807
use BTreeMap instead of HashMap
bcantrill May 14, 2023
f5d03e1
Elide duplicate unencrypted contexts
jmpesp May 15, 2023
ef32020
Merge branch 'elide_duplicate_unencrypted_contexts' of github.com:jmp…
bcantrill May 15, 2023
ac66660
Elide duplicate unencrypted contexts
jmpesp May 15, 2023
79e27f0
fix "crutest dep" test failure
bcantrill May 15, 2023
092058c
`measure-iops` should submit different blocks
jmpesp May 15, 2023
e6592e5
fix test_reconnect.sh test failure
bcantrill May 18, 2023
f0df895
rustfmt
bcantrill May 18, 2023
b1fda0d
Merge branch 'main' into perfwork
bcantrill May 18, 2023
023d326
Merge branch 'perfwork' of github.com:oxidecomputer/crucible into per…
bcantrill May 27, 2023
9b0f76c
Merge branch 'main' of github.com:oxidecomputer/crucible
bcantrill May 27, 2023
d68b0b1
Merge branch 'main' into perfwork
bcantrill May 27, 2023
784f845
fix build
bcantrill May 28, 2023
e4ade77
allow flush interval to be tuned + change default
bcantrill May 17, 2023
ecd8ccd
build
bcantrill May 28, 2023
5734060
use mem::take instead of drain
faithanalog May 27, 2023
87ae64c
use INTEGER (64 bits) for hash/on_disk_hash
faithanalog May 27, 2023
e4a2f4b
downstairs sqlite: use unix-excl VFS, removing SHM file
faithanalog May 27, 2023
a0daa8b
use WITHOUT ROWID for the contexts table
faithanalog May 27, 2023
b2df7fc
fix the test suite
faithanalog May 28, 2023
98215b7
meant to get this cleanup in there
faithanalog May 28, 2023
ceaf584
Revert "meant to get this cleanup in there"
bcantrill May 28, 2023
bc69933
Revert "use WITHOUT ROWID for the contexts table"
bcantrill May 28, 2023
eab9117
Revert "downstairs sqlite: use unix-excl VFS, removing SHM file"
bcantrill May 28, 2023
244795a
Revert "use INTEGER (64 bits) for hash/on_disk_hash"
bcantrill May 28, 2023
1e90197
Revert "use mem::take instead of drain"
bcantrill May 28, 2023
a97fc2d
fix panic + reconstruction failures
bcantrill May 29, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
avoid some scans
  • Loading branch information
bcantrill committed May 13, 2023
commit c21b262ebe629a834e0557e21bc0013516f7ee79
35 changes: 11 additions & 24 deletions upstairs/src/lib.rs
Original file line number Diff line number Diff line change
@@ -2475,6 +2475,8 @@ struct Downstairs {
ds_last_flush: Vec<u64>,
downstairs_errors: HashMap<u8, u64>, // client id -> errors
ds_active: HashMap<u64, DownstairsIO>,
ds_new: Vec<Vec<u64>>,

next_id: u64,
completed: AllocRingBuffer<u64>,
completed_jobs: AllocRingBuffer<WorkSummary>,
@@ -2551,6 +2553,7 @@ impl Downstairs {
ds_last_flush: vec![0; 3],
downstairs_errors: HashMap::new(),
ds_active: HashMap::new(),
ds_new: vec![Vec::new(); 3],
completed: AllocRingBuffer::with_capacity(2048),
completed_jobs: AllocRingBuffer::with_capacity(8),
next_id: 1000,
@@ -2994,45 +2997,28 @@ impl Downstairs {
* Return a list of downstairs request IDs that represent unissued
* requests for this client.
*/
fn new_work(&self, client_id: u8) -> Vec<u64> {
self.ds_active
.values()
.filter_map(|job| {
if let Some(IOState::New) = job.state.get(&client_id) {
Some(job.ds_id)
} else {
None
}
})
.collect()
fn new_work(&mut self, client_id: u8) -> Vec<u64> {
self.ds_new[client_id as usize].drain(..).collect()
}

/**
* Return a count of downstairs request IDs of work we have sent
* for this client, but don't yet have a response.
*/
fn submitted_work(&self, client_id: u8) -> usize {
self.ds_active
.values()
.filter(|job| {
Some(&IOState::InProgress) == job.state.get(&client_id)
})
.count()
self.io_state_count.in_progress[client_id as usize] as usize
}

/**
* Return a count of downstairs request IDs of total new and submitted
* work we have for a downstairs.
*/
fn total_live_work(&self, client_id: u8) -> usize {
self.ds_active
.values()
.filter(|job| {
Some(&IOState::InProgress) == job.state.get(&client_id)
|| Some(&IOState::New) == job.state.get(&client_id)
})
.count()
(self.io_state_count.new[client_id as usize]
+ self.io_state_count.in_progress[client_id as usize])
as usize
}

/**
* Build a list of jobs that are ready to be acked.
*/
@@ -3076,6 +3062,7 @@ impl Downstairs {
}
_ => {
self.io_state_count.incr(&IOState::New, cid);
self.ds_new[cid as usize].push(io.ds_id);
}
}
}