| Age | Commit message (Collapse) | Author |
|---|
|
* include/mach_debug/slab_info.h (CACHE_FLAGS_NO_CPU_POOL,
CACHE_FLAGS_SLAB_EXTERNAL, CACHE_FLAGS_NO_RECLAIM,
CACHE_FLAGS_VERIFY, CACHE_FLAGS_DIRECT): Remove macros.
* kern/slab.c (host_slab_info): Pass raw cache flags to caller.
|
|
The slab allocator has grown to use multiple ways to allocate slabs
as well as track them, which got a little messy. One consequence is
the breaking of the KMEM_CF_VERIFY option. In order to make the code
less confusing, this change expresses all options as explicit cache
flags and clearly defines their relationships.
The special kmem_slab and vm_map_entry caches are initialized
accordingly.
* kern/slab.c (KMEM_CF_DIRECTMAP): Rename to ...
(KMEM_CF_PHYSMEM): ... this new macro.
(KMEM_CF_DIRECT): Restore macro.
(KMEM_CF_USE_TREE, KMEM_CF_USE_PAGE): New macros.
(KMEM_CF_VERIFY): Update value.
(kmem_pagealloc_directmap): Rename to...
(kmem_pagealloc_physmem): ... this new function.
(kmem_pagefree_directmap): Rename to ...
(kmem_pagefree_physmem): ... this new function.
(kmem_pagealloc, kmem_pagefree): Update macro names.
(kmem_slab_use_tree): Remove function.
(kmem_slab_create, kmem_slab_destroy): Update according to the new
cache flags.
(kmem_cache_compute_sizes): Rename to ...
(kmem_cache_compute_properties): ... this new function, and update
to properly set cache flags.
(kmem_cache_init): Update call to kmem_cache_compute_properties.
(kmem_cache_alloc_from_slab): Check KMEM_CF_USE_TREE instead of
calling the defunct kmem_slab_use_tree function.
(kmem_cache_free_to_slab): Update according to the new cache flags.
kmem_cache_free_verify): Add assertion.
(slab_init): Update initialization of kmem_slab_cache.
* kern/slab.h (KMEM_CACHE_DIRECTMAP): Rename to ...
(KMEM_CACHE_PHYSMEM): ... this new macro.
* vm/vm_map.c (vm_map_init): Update initialization of vm_map_entry_cache.
|
|
Caches that use external slab data but allocate slabs from the direct
physical mapping can look up slab data in constant time by associating
the slab data directly with the underlying page.
* kern/slab.c (kmem_slab_use_tree): Take KMEM_CF_DIRECTMAP into account.
(kmem_slab_create): Set page private data if relevant.
(kmem_slab_destroy): Clear page private data if relevant.
(kmem_cache_free_to_slab): Use page private data if relevant.
* vm/vm_page.c (vm_page_init_pa): Set `priv' member to NULL.
* vm/vm_page.h (vm_page_set_priv, vm_page_get_priv): New functions.
|
|
* kern/slab.c (slab_init): Remove unused variables.
|
|
Since the slab allocator has been changed to sit directly on top of the
physical allocator, failures caused by fragmentation have been observed,
as one could expect. This change makes the slab allocator revert to
kernel virtual memory when allocating larger-than-page slabs. This
solution is motivated in part to avoid the complexity of other solutions
such as page mobility, and also because a microkernel cannot be extended
to new arbitrary uncontrolled usage patterns such as a monolithic kernel
with loadable modules. As such, large objects are rare, and their use
infrequent, which is compatible with the use of kernel virtual memory.
* kern/slab.c: Update module description.
(KMEM_CF_SLAB_EXTERNAL, KMEM_CF_VERIFY): Update values.
(KMEM_CF_DIRECT): Remove macro.
(KMEM_CF_DIRECTMAP): New macro.
(kmem_pagealloc_directmap, kmem_pagefree_directmap,
kmem_pagealloc_virtual, kmem_pagefree_virtual): New functions.
(kmem_pagealloc, kmem_pagefree, kmem_slab_create, kmem_slab_destroy,
kalloc, kfree): Update to use the new pagealloc functions.
(kmem_cache_compute_sizes): Update the algorithm used to determine slab
size and other cache properties.
(kmem_slab_use_tree, kmem_cache_free_to_slab, host_slab_info): Update to
correctly use the cache flags.
(slab_init): Add KMEM_CACHE_DIRECTMAP to the kmem_slab_cache init flags.
* kern/slab.h (KMEM_CACHE_VERIFY): Change value.
(KMEM_CACHE_DIRECTMAP): New macro.
* vm/vm_map.c (vm_map_init): Add KMEM_CACHE_DIRECTMAP to the
vm_map_entry_cache init flags.
|
|
Now that the slab allocator doesn't use kernel virtual memory any more,
this map has become irrelevant.
* kern/slab.c (KMEM_MAP_SIZE): Remove macro.
(kmem_map_store, kmem_map): Remove variables.
(slab_init): Remove call kmem_submap.
* kern/slab.h (kmem_map): Remove extern declaration.
|
|
Allocating directly out of the physical memory allocator makes the slab
allocator vulnerable to failures due to fragmentation. This change makes
the slab allocator use the lowest possible size for its slabs to reduce
the chance of contiguous allocation failures.
* kern/slab.c (KMEM_MIN_BUFS_PER_SLAB, KMEM_SLAB_SIZE_THRESHOLD): Remove
macros.
(kmem_cache_compute_sizes): Update the algorithm used to determine slab
size and other cache properties.
|
|
A few errors were introduced in the latest changes.
o Add VM_PAGE_WAIT calls around physical allocation attempts in case of
memory exhaustion.
o Fix stack release.
o Fix memory exhaustion report.
o Fix free page accounting.
* kern/slab.c (kmem_pagealloc, kmem_pagefree): New functions
(kmem_slab_create, kmem_slab_destroy, kalloc, kfree): Use kmem_pagealloc
and kmem_pagefree instead of the raw page allocation functions.
(kmem_cache_compute_sizes): Don't store slab order.
* kern/slab.h (struct kmem_cache): Remove `slab_order' member.
* kern/thread.c (stack_alloc): Call VM_PAGE_WAIT in case of memory
exhaustion.
(stack_collect): Call vm_page_free_contig instead of kmem_free to
release pages.
* vm/vm_page.c (vm_page_seg_alloc): Fix memory exhaustion report.
(vm_page_setup): Don't update vm_page_free_count.
(vm_page_free_pa): Check page parameter.
(vm_page_mem_free): New function.
* vm/vm_page.h (vm_page_free_count): Remove extern declaration.
(vm_page_mem_free): New prototype.
* vm/vm_pageout.c: Update comments not to refer to vm_page_free_count.
(vm_pageout_scan, vm_pageout_continue, vm_pageout): Use vm_page_mem_free
instead of vm_page_free_count, update types accordingly.
* vm/vm_resident.c (vm_page_free_count, vm_page_free_count_minimum):
Remove variables.
(vm_page_free_avail): New variable.
(vm_page_bootstrap, vm_page_grab, vm_page_release, vm_page_grab_contig,
vm_page_free_contig, vm_page_wait): Use vm_page_mem_free instead of vm_page_free_count,
update types accordingly, don't set vm_page_free_count_minimum.
* vm/vm_user.c (vm_statistics): Likewise.
|
|
In order to increase the amount of memory available for kernel objects,
without reducing the amount of memory available for user processes,
a new allocation strategy is introduced in this change.
Instead of allocating kernel objects out of kernel virtual memory,
the slab allocator directly uses the direct mapping of physical
memory as its backend. This largely increases the kernel heap, and
removes the need for address translation updates.
In order to allow this strategy, an assumption made by the interrupt
code had to be removed. In addition, kernel stacks are now also
allocated directly from the physical allocator.
* i386/i386/db_trace.c: Include i386at/model_dep.h
(db_i386_reg_value): Update stack check.
* i386/i386/locore.S (trap_from_kernel, all_intrs,
int_from_intstack): Update interrupt handling.
* i386/i386at/model_dep.c: Include kern/macros.h.
(int_stack, int_stack_base): New variables.
(int_stack_high): Remove variable.
(i386at_init): Update interrupt stack initialization.
* i386/i386at/model_dep.h: Include i386/vm_param.h.
(int_stack_top, int_stack_base): New extern declarations.
(ON_INT_STACK): New macro.
* kern/slab.c: Include vm/vm_page.h
(KMEM_CF_NO_CPU_POOL, KMEM_CF_NO_RECLAIM): Remove macros.
(kmem_pagealloc, kmem_pagefree, kalloc_pagealloc, kalloc_pagefree): Remove
functions.
(kmem_slab_create): Allocate slab pages directly from the physical allocator.
(kmem_slab_destroy): Release slab pages directly to the physical allocator.
(kmem_cache_compute_sizes): Update the slab size computation algorithm to
return a power-of-two suitable for the physical allocator.
(kmem_cache_init): Remove custom allocation function pointers.
(kmem_cache_reap): Remove check on KMEM_CF_NO_RECLAIM.
(slab_init, kalloc_init): Update calls to kmem_cache_init.
(kalloc, kfree): Directly fall back on the physical allocator for big
allocation sizes.
(host_slab_info): Remove checks on defunct flags.
* kern/slab.h (kmem_slab_alloc_fn_t, kmem_slab_free_fn_t): Remove types.
(struct kmem_cache): Add `slab_order' member, remove `slab_alloc_fn' and
`slab_free_fn' members.
(KMEM_CACHE_NOCPUPOOL, KMEM_CACHE_NORECLAIM): Remove macros.
(kmem_cache_init): Update prototype, remove custom allocation functions.
* kern/thread.c (stack_alloc): Allocate stacks from the physical allocator.
* vm/vm_map.c (vm_map_kentry_cache, kentry_data, kentry_data_size): Remove
variables.
(kentry_pagealloc): Remove function.
(vm_map_init): Update calls to kmem_cache_init, remove initialization of
vm_map_kentry_cache.
(vm_map_create, _vm_map_entry_dispose, vm_map_copyout): Unconditionnally
use vm_map_entry_cache.
* vm/vm_map.h (kentry_data, kentry_data_size, kentry_count): Remove extern
declarations.
* vm/vm_page.h (VM_PT_STACK): New page type.
* device/dev_lookup.c (dev_lookup_init): Update calls to kmem_cache_init.
* device/dev_pager.c (dev_pager_hash_init, device_pager_init): Likewise.
* device/ds_routines.c (mach_device_init, mach_device_trap_init): Likewise.
* device/net_io.c (net_io_init): Likewise.
* i386/i386/fpu.c (fpu_module_init): Likewise.
* i386/i386/machine_task.c (machine_task_module_init): Likewise.
* i386/i386/pcb.c (pcb_module_init): Likewise.
* i386/intel/pmap.c (pmap_init): Likewise.
* ipc/ipc_init.c (ipc_bootstrap): Likewise.
* ipc/ipc_marequest.c (ipc_marequest_init): Likewise.
* kern/act.c (global_act_init): Likewise.
* kern/processor.c (pset_sys_init): Likewise.
* kern/rdxtree.c (rdxtree_cache_init): Likewise.
* kern/task.c (task_init): Likewise.
* vm/memory_object_proxy.c (memory_object_proxy_init): Likewise.
* vm/vm_external.c (vm_external_module_initialize): Likewise.
* vm/vm_fault.c (vm_fault_init): Likewise.
* vm/vm_object.c (vm_object_bootstrap): Likewise.
* vm/vm_resident.c (vm_page_module_init): Likewise.
(vm_page_bootstrap): Remove initialization of kentry_data.
|
|
To avoid running out of memory due to the increased consumption by radix
trees which replaced arrays.
* kern/slab.c (KMEM_MAP_SIZE): Bump from 96MiB to 128MiB.
* i386/i386/vm_param.h (VM_KERNEL_MAP_SIZE): Add 32MiB accordingly.
|
|
* kern/slab.c (_slab_info): Print total used and reclaimable memory.
|
|
* kern/slab.c (_slab_info): Include flags in the slab information.
|
|
* ddb/db_command.c (db_show_cmds): Add `slabinfo'.
* kern/slab.c (slab_info): Generalize so that it can be used with
different printf-like functions, and turn it into a static function.
(slab_info): New wrapper retaining the old behaviour.
(db_show_slab_info): New wrapper that uses `db_printf' instead.
* kern/slab.h (db_show_slab_info): New declaration.
|
|
* kern/slab.c (host_slab_info): Fix locking.
|
|
Import the macro definitions from the x15 kernel project, and replace
all similar definitions littered all over the place with it.
Importing this file will make importing code from the x15 kernel
easier. We are already using the red-black tree implementation and
the slab allocator from it, and we will import even more code in the
near future.
* kern/list.h: Do not define `structof', include `macros.h' instead.
* kern/rbtree.h: Likewise.
* kern/slab.c: Do not define `ARRAY_SIZE', include `macros.h' instead.
* i386/grub/misc.h: Likewise.
* i386/i386/xen.h: Do not define `barrier', include `macros.h' instead.
* kern/macro_help.h: Delete file. Replaced by `macros.h'.
* kern/macros.h: New file.
* Makefrag.am (libkernel_a_SOURCES): Add new file, remove old file.
* device/dev_master.h: Adopt accordingly.
* device/io_req.h: Likewise.
* device/net_io.h: Likewise.
* i386/intel/read_fault.c: Likewise.
* ipc/ipc_kmsg.h: Likewise.
* ipc/ipc_mqueue.h: Likewise.
* ipc/ipc_object.h: Likewise.
* ipc/ipc_port.h: Likewise.
* ipc/ipc_space.h: Likewise.
* ipc/ipc_splay.c: Likewise.
* ipc/ipc_splay.h: Likewise.
* kern/assert.h: Likewise.
* kern/ast.h: Likewise.
* kern/pc_sample.h: Likewise.
* kern/refcount.h: Likewise.
* kern/sched.h: Likewise.
* kern/sched_prim.c: Likewise.
* kern/timer.c: Likewise.
* kern/timer.h: Likewise.
* vm/vm_fault.c: Likewise.
* vm/vm_map.h: Likewise.
* vm/vm_object.h: Likewise.
* vm/vm_page.h: Likewise.
|
|
* kern/slab.c (kmem_cache_compute_sizes): Initialize optimal_size and
assert that a size is selected.
|
|
* kern/slab.c (KMEM_MAP_SIZE): Decrease from 128MiB to 96MiB.
|
|
The slab allocator relies on the fact that kmem_cache_error does not
return. Previously, kmem_error was using printf. Use panic instead.
Found using the Clang Static Analyzer.
* kern/slab.c (kmem_error): Use panic instead of printf.
|
|
* kern/slab.c (kmem_cache_error): Use kmem_warn instead of kmem_error
to print the cache name and its address.
|
|
info_size is initialized to a random value. Quiet the warning by
initializing to zero.
* kern/slab.c (info_size): Initialize to zero.
|
|
optimal_embed is initialized to a random value. Quiet the warning by
initializing to zero.
* kern/slab.c (optimal_embed): Initialize to zero.
|
|
* kern/slab.c (slab_info): Fix format for vm_size_t.
|
|
There is currently no actual use of constructors, which is why this bug has
been long overlooked.
* kern/slab.c (kmem_cpu_pool_fill): Call constructor on buffers unless
verification is enabled for the cache, or the constructor is NULL.
|
|
* kern/slab.c (kmem_cache_free): Relock cache before retrying releasing
an object to the CPU pool layer.
|
|
This reverts a change brought when reworking slab lists handling that made
the allocator store slabs in LIFO order, whatever their reference count.
While it's fine for free slabs, it actually increased fragmentation for
partial slabs.
* kern/slab.c (kmem_cache_alloc_from_slab): Insert slabs that become partial
at the end of the partial slabs list.
|
|
This change increases clarity.
* kern/list.h (list_insert): Rename to ...
(list_insert_head): ... this.
* kern/slab.c: Update calls to list_insert.
|
|
* kern/slab.c (kalloc_init): Remove unused variables.
|
|
Instead of walking the list of free slabs while holding the cache lock,
detach the list from the cache and directly compute the final count values,
and destroy slabs after releasing the cache lock.
* kern/slab.c (kmem_cache_reap): Optimize.
|
|
Don't enforce strong ordering of partial slabs. Separating partial slabs
from free slabs is already effective against fragmentation, and sorting
would sometimes cause pathological scalability issues. In addition, store
new slabs (whether free or partial) in LIFO order for better cache usage.
* kern/slab.c (kmem_cache_grow): Insert new slab at the head of the slabs list.
(kmem_cache_alloc_from_slab): Likewise. In addition, don't sort partial slabs.
(kmem_cache_free_to_slab): Likewise.
* kern/slab.h: Remove comment about partial slabs sorting.
|
|
* kern/slab.c (kmem_cache_free): Lock cache before releasing an object to
the slab layer.
|
|
This problem was overlooked because of simple locks being no-ops.
* kern/slab.c (slab_collect): Fix lock name to kmem_cache_list_lock.
(host_slab_info): Likewise.
|
|
The purpose of this function is to allow kernel code to display the state
of the slab caches in situations where the host_slab_info RPC wouldn't be
available, e.g. before a panic.
* kern/slab.c (slab_info): New function.
* kern/slab.h: Add declaration for slab_info.
|
|
The slab garbage collector uses sched_tick as its time reference, which
is increased every seconds, while the interval is expressed in clock
ticks. Use the proper time reference instead.
* kern/slab.c (kmem_gc_last_tick): Declare as unsigned long.
(slab_collect): Use elapsed_ticks instead of sched_tick.
|
|
* kern/slab.c (KMEM_GC_INTERVAL): Increase to 5 seconds.
|
|
* ipc/ipc_table.c: Add #include <kern/slab.h>.
(ipc_table_alloc): Use kmem_map instead of kalloc_map when allocating
a table.
(ipc_table_realloc): Likewise for reallocation.
(ipc_table_free): Likewise for release.
* kern/kalloc.h (kalloc_map): Remove declaration.
* kern/slab.c (KMEM_MAP_SIZE): Increase to 128 MiB.
(KALLOC_MAP_SIZE): Remove macro.
(kalloc_map_store): Remove variable.
(kalloc_map): Likewise.
(kalloc_pagealloc): Use kmem_map instead of kalloc_map for general
purpose allocations.
(kalloc_pagefree): Likewise.
(kalloc_init): Remove the creation of kalloc_map.
|
|
Maksym and I have assigned copyright to the Free Software Foundation.
In addition, restore the original upstream copyrights for correctness.
* kern/list.h: Fix copyright assignment.
* kern/rbtree.c: Likewise.
* kern/rbtree.h: Likewise.
* kern/rbtree_i.h: Likewise.
* kern/slab.c: Likewise.
* kern/slab.h: Likewise.
|
|
TODO: remonter formats
* i386/include/mach/i386/vm_types.h (vm_offset_t): Define to unsigned long.
(signed32_t): Define to signed int.
(unsigned32_t): Define to unsigned int.
* i386/include/mach/sa/stdarg.h (__va_size): Use sizeof(unsigned long)-1
instead of 3.
* include/mach/port.h (mach_port_t): Define to vm_offset_t instead of
natural_t.
* include/sys/types.h (size_t): Define to unsigned long instead of
natural_t.
* linux/src/include/asm-i386/posix_types.h (__kernel_size_t): Define to
unsigned long.
(__kernel_ssize_t): Define to long.
* linux/src/include/linux/stddef.h (size_t): Define to unsigned long.
* device/dev_pager.c (dev_pager_hash): Cast port to vm_offset_t insted of
natural_t.
(device_pager_data_request): Fix format.
* device/ds_routines.c (ds_no_senders): Fix format.
* i386/i386/io_map.c (io_map): Likewise.
* i386/i386at/autoconf.c (take_dev_irq): Likewise.
* i386/i386at/com.c (comattach): Likewise.
* i386/i386at/lpr.c (lprattach): Likewise.
* i386/i386at/model_dep.c (mem_size_init, mem_size_init, c_boot_entry):
Likewise.
* i386/intel/pmap.c (pmap_enter): Likewise.
* ipc/ipc_notify.c (ipc_notify_port_deleted, ipc_notify_msg_accepted,
ipc_notify_dead_name): Likewise.
* ipc/mach_port.c (mach_port_destroy, mach_port_deallocate): Likewise.
* kern/ipc_kobject.c (ipc_kobject_destroy): Likewise.
* kern/slab.c (kalloc_init): Likewise.
* vm/vm_fault.c (vm_fault_page): Likewise.
* vm/vm_map.c (vm_map_pmap_enter): Likewise.
* xen/block.c (device_read): Likewise.
* device/net_io.c (bpf_match): Take unsigned long * instead of unsigned int
*.
(bpf_do_filter): Make mem unsigned long instead of long.
* i386/i386/ktss.c (ktss_init): Cast pointer to unsigned long instead of
unsigned.
* i386/i386/pcb.c (stack_attach, switch_ktss): Cast pointers to long instead of
int.
* i386/i386/trap.c (dump_ss): Likewise.
* ipc/ipc_hash.c (IH_LOCAL_HASH): Cast object to vm_offset_t.
* ipc/mach_msg.c (mach_msg_receive, mach_msg_receive_continue): Cast kmsg to
vm_offset_t instead of natural_t.
* kern/pc_sample.c (take_pc_sample): Cast to vm_offset_t instead of
natural_t.
* kern/boot_script.c (sym, arg): Set type of `val' field to long instead of int.
(create_task, builtin_symbols, boot_script_parse_line,
boot_script_define_function): Cast to long instead of int.
* kern/bootstrap.c (bootstrap_create): Likewise.
* kern/sched_prim.c (decl_simple_lock_data): Likewise.
* kern/printf.c (vsnprintf): Set size type to size_t.
* kern/printf.h (vsnprintf): Likewise.
* vm/vm_map.h (kentry_data_size): Fix type to vm_size_t.
* vm/vm_object.c (vm_object_pmap_protect_by_page): Fix size parameter type
to vm_size_t.
|
|
As it is intended to completely replace the zone allocator, remove it on
the way. So long to the venerable code !
* Makefrag.am (libkernel_a_SOURCES): Add kern/slab.{c,h}, remove kern/kalloc.c
and kern/zalloc.{c,h}.
* configfrag.ac (SLAB_VERIFY, SLAB_USE_CPU_POOLS): Add defines.
* i386/Makefrag.am (libkernel_a_SOURCES): Remove i386/i386/zalloc.h.
* i386/configfrag.ac (CPU_L1_SHIFT): Remove define.
* include/mach_debug/slab_info.h: New file.
* kern/slab.c: Likewise.
* kern/slab.h: Likewise.
* i386/i386/zalloc.h: Remove file.
* include/mach_debug/zone_info.h: Likewise.
* kern/kalloc.c: Likewise.
* kern/zalloc.c: Likewise.
* kern/zalloc.h: Likewise.
|
