Benchmarks

Here, we collect some simple benchmarks of BSeries.jl. Take them with a grain of salt since they run on virtual machines in the cloud to generate the documentation automatically. You can of course also copy the code and run the benchmarks locally yourself.

Comparing different symbolic packages

Symbolic computations of modified_equations and modifying_integrators in BSeries.jl support

• SymEngine.jl,
• SymPy.jl,
• SymPyPythonCall.jl, and
• Symbolics.jl

as symbolic backends. Here, we compare them in the context of the explicit midpoint method and the nonlinear oscillator ODE

\[u'(t) = \frac{1}{\| u(t) \|^2} \begin{pmatrix} -u_2(t) \\ u_1(t) \end{pmatrix}.\]

This particular combination of explicit Runge-Kutta method and ODE is special since the explicit midpoint method is unconditionally energy-conserving for this problem^{[RanochaKetcheson2020]}.

First, we set up some code to perform the benchmarks. Here, we use a very naive approach, run the code twice (to see the effect of compilation) and use @time to print the runtime. More sophisticated approaches should of course use something like @benchmark from BenchmarkTools.jl. However, this simple and cheap version suffices to compare the orders of magnitude.

using BSeries, StaticArrays

function benchmark(u, dt, subs, order)
  # explicit midpoint method
  A = @SArray [0 0; 1//2 0]
  b = @SArray [0, 1//1]
  c = @SArray [0, 1//2]

  # nonlinear oscillator
  f = [-u[2], u[1]] / (u[1]^2 + u[2]^2)

  println("\n Computing the series coefficients:")
  @time coefficients = modifying_integrator(A, b, c, order)
  @time coefficients = modifying_integrator(A, b, c, order)

  println("\n Computing the series including elementary differentials:")
  @time series = modifying_integrator(f, u, dt, A, b, c, order)
  @time series = modifying_integrator(f, u, dt, A, b, c, order)

  substitution_variables = Dict(u[1] => 1//1, u[2] => 0//1)

  println("\n Substituting the initial condition:")
  @time subs.(series, (substitution_variables, ))
  @time subs.(series, (substitution_variables, ))

  println("\n")
end

benchmark (generic function with 1 method)

Next, we load the symbolic packages and run the benchmarks.

using SymEngine: SymEngine
using SymPyPythonCall: SymPyPythonCall
using Symbolics: Symbolics

println("SymEngine")
dt   = SymEngine.symbols("dt")
u    = SymEngine.symbols("u1, u2")
subs = SymEngine.subs
benchmark(u, dt, subs, 8)

println("SymPy")
dt   = SymPyPythonCall.symbols("dt")
u    = SymPyPythonCall.symbols("u1, u2")
subs = SymPyPythonCall.subs
benchmark(u, dt, subs, 8)

println("Symbolics")
Symbolics.@variables dt
u = Symbolics.@variables u1 u2
subs = Symbolics.substitute
benchmark(u, dt, subs, 8)

SymEngine

 Computing the series coefficients:
  0.005625 seconds (242 allocations: 95.047 KiB)
  0.005623 seconds (242 allocations: 95.047 KiB)

 Computing the series including elementary differentials:
  0.999573 seconds (891.85 k allocations: 57.797 MiB, 1.34% gc time, 98.33% compilation time)
  0.017180 seconds (47.59 k allocations: 1.075 MiB)

 Substituting the initial condition:
  0.067156 seconds (114.31 k allocations: 7.050 MiB, 93.04% compilation time)
  0.004197 seconds (12.74 k allocations: 138.953 KiB)


SymPy

 Computing the series coefficients:
  0.005642 seconds (242 allocations: 95.047 KiB)
  0.005617 seconds (242 allocations: 95.047 KiB)

 Computing the series including elementary differentials:
  2.223194 seconds (694.78 k allocations: 46.303 MiB, 44.25% compilation time)
  0.802681 seconds (25.50 k allocations: 874.117 KiB)

 Substituting the initial condition:
  0.752662 seconds (191.03 k allocations: 13.065 MiB, 18.26% compilation time)
  0.618918 seconds (161 allocations: 3.641 KiB)


Symbolics

 Computing the series coefficients:
  0.005638 seconds (242 allocations: 95.047 KiB)
  0.005606 seconds (242 allocations: 95.047 KiB)

 Computing the series including elementary differentials:
  4.750111 seconds (6.07 M allocations: 381.344 MiB, 2.17% gc time, 84.84% compilation time)
  1.016258 seconds (3.61 M allocations: 210.936 MiB, 33.64% gc time)

 Substituting the initial condition:
  7.416752 seconds (21.09 M allocations: 804.034 MiB, 1.34% gc time, 35.49% compilation time)
  4.714534 seconds (18.94 M allocations: 664.667 MiB, 1.23% gc time)

These results were obtained using the following versions.

using InteractiveUtils
versioninfo()

using Pkg
Pkg.status(["BSeries", "RootedTrees", "SymEngine", "SymPyPythonCall", "Symbolics"],
           mode=PKGMODE_MANIFEST)

Julia Version 1.10.4
Commit 48d4fd48430 (2024-06-04 10:41 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 4 × AMD EPYC 7763 64-Core Processor
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, znver3)
Threads: 1 default, 0 interactive, 1 GC (on 4 virtual cores)
Environment:
  JULIA_PKG_SERVER_REGISTRY_PREFERENCE = eager
  LD_LIBRARY_PATH = /opt/hostedtoolcache/Python/3.9.19/x64/lib
  JULIA_PYTHONCALL_EXE = /home/runner/work/BSeries.jl/BSeries.jl/docs/.CondaPkg/env/bin/python
Status `~/work/BSeries.jl/BSeries.jl/docs/Manifest.toml`
  [ebb8d67c] BSeries v0.1.64 `~/work/BSeries.jl/BSeries.jl`
  [47965b36] RootedTrees v2.23.0
⌃ [123dc426] SymEngine v0.10.0
⌃ [bc8888f7] SymPyPythonCall v0.3.0
  [0c5d862f] Symbolics v5.35.1
Info Packages marked with ⌃ have new versions available and may be upgradable.

Comparison with other packages

There are also other open source packages for B-series. Currently, we are aware of the Python packages

• BSeries
• pybs
• orderconditions

If you know about similar open source packages out there, please inform us, e.g., by creating an issue on GitHub.

The packages listed above and BSeries.jl all use different approaches and have different features. Thus, comparisons must be restricted to their common subset of features. Here, we present some simple performance comparisons. Again, we just use (the equivalent of) @time twice to get an idea of the performance after compilation, allowing us to compare orders of magnitude.

Python package BSeries

First, we start with the Python package BSeries and the following benchmark script.

import sys
from importlib.metadata import version
print("Python version", sys.version)
print("Package version", version('pybs'))

import time
import BSeries.bs as bs
import nodepy.runge_kutta_method as rk

midpoint_method = rk.loadRKM("Mid22")
up_to_order = 9


print("\nModified equation")

start_time = time.time()
series = bs.modified_equation(None, None,
                              midpoint_method.A, midpoint_method.b,
                              up_to_order, True)
result = sum(series.values())
end_time = time.time()
print(result)
print("", end_time - start_time, "seconds")

start_time = time.time()
series = bs.modified_equation(None, None,
                              midpoint_method.A, midpoint_method.b,
                              up_to_order, True)
result = sum(series.values())
end_time = time.time()
print(result)
print("", end_time - start_time, "seconds")


print("\nModifying integrator")

start_time = time.time()
series = bs.modifying_integrator(None, None,
                                 midpoint_method.A, midpoint_method.b,
                                 up_to_order, True)
result = sum(series.values())
end_time = time.time()
print(result)
print("", end_time - start_time, "seconds")

start_time = time.time()
series = bs.modifying_integrator(None, None,
                                 midpoint_method.A, midpoint_method.b,
                                 up_to_order, True)
result = sum(series.values())
end_time = time.time()
print(result)
print("", end_time - start_time, "seconds")

The results are as follows.

Python version 3.9.19 (main, Jul 16 2024, 19:03:54) 
[GCC 11.4.0]
Package version 0.3

Modified equation
19063/26880
 14.225465297698975 seconds
19063/26880
 13.764608383178711 seconds

Modifying integrator
5460293/241920
 12.636103868484497 seconds
5460293/241920
 11.868774175643921 seconds

Python package pybs

Next, we look at the Python package pybs and the following benchmark script. Note that this package does not provide functionality for modifying integrators.

import sys
from importlib.metadata import version
print("Python version", sys.version)
print("Package version", version('pybs'))

import time
import pybs
from pybs.rungekutta import methods as rk_methods

midpoint_method = rk_methods.RKmidpoint
up_to_order = 9
number_of_terms = pybs.unordered_tree.number_of_trees_up_to_order(up_to_order+1)

from itertools import islice
def first_values(f, n):
  return (f(tree) for tree in islice(pybs.unordered_tree.tree_generator(), 0, n))


print("\nModified equation")

start_time = time.time()
midpoint_series = midpoint_method.phi()
series = pybs.series.modified_equation(midpoint_series)
result = sum(first_values(series, number_of_terms))
end_time = time.time()
print(result)
print("", end_time - start_time, "seconds")

start_time = time.time()
midpoint_series = midpoint_method.phi()
series = pybs.series.modified_equation(midpoint_series)
result = sum(first_values(series, number_of_terms))
end_time = time.time()
print(result)
print("", end_time - start_time, "seconds")


print("\nEnergy preservation")

start_time = time.time()
a = pybs.series.AVF
b = pybs.series.modified_equation(a)
result = pybs.series.energy_preserving_upto_order(b, up_to_order)
end_time = time.time()
print(result)
print("", end_time - start_time, "seconds")

start_time = time.time()
a = pybs.series.AVF
b = pybs.series.modified_equation(a)
result = pybs.series.energy_preserving_upto_order(b, up_to_order)
end_time = time.time()
print(result)
print("", end_time - start_time, "seconds")


print("\nSymplecticity (conservation of quadratic invariants)")

start_time = time.time()
a = rk_methods.RKimplicitMidpoint.phi()
result = pybs.series.symplectic_up_to_order(a, up_to_order)
end_time = time.time()
print(result)
print("", end_time - start_time, "seconds")

start_time = time.time()
a = rk_methods.RKimplicitMidpoint.phi()
result = pybs.series.symplectic_up_to_order(a, up_to_order)
end_time = time.time()
print(result)
print("", end_time - start_time, "seconds")

The results are as follows.

Python version 3.9.19 (main, Jul 16 2024, 19:03:54) 
[GCC 11.4.0]
Package version 0.3

Modified equation
19063/26880
 5.590765476226807 seconds
19063/26880
 5.537620782852173 seconds

Energy preservation
9
 5.730681657791138 seconds
9
 5.538189172744751 seconds

Symplecticity (conservation of quadratic invariants)
9
 0.03379034996032715 seconds
9
 0.019370317459106445 seconds

Python package orderconditions

Next, we look at the Python package orderconditions of Valentin Dallerit and the following benchmark script.

import sys
print("Python version", sys.version)

import time
from orderConditions import BSeries
import nodepy.runge_kutta_method as rk

midpoint_method = rk.loadRKM("Mid22")
up_to_order = 9


print("\nModified equation")

start_time = time.time()
BSeries.set_order(up_to_order)
Y1 = BSeries.y() + midpoint_method.A[1,0] * BSeries.hf()
rk2 = BSeries.y() + midpoint_method.b[0] * BSeries.hf() + midpoint_method.b[1] * BSeries.compo_hf(Y1)
series = BSeries.modified_equation(rk2)
result = series.sum()
end_time = time.time()
print(result)
print("", end_time - start_time, "seconds")

start_time = time.time()
BSeries.set_order(up_to_order)
Y1 = BSeries.y() + midpoint_method.A[1,0] * BSeries.hf()
rk2 = BSeries.y() + midpoint_method.b[0] * BSeries.hf() + midpoint_method.b[1] * BSeries.compo_hf(Y1)
series = BSeries.modified_equation(rk2)
result = series.sum()
end_time = time.time()
print(result)
print("", end_time - start_time, "seconds")


print("\nModifying integrator")

start_time = time.time()
BSeries.set_order(up_to_order)
Y1 = BSeries.y() + midpoint_method.A[1,0] * BSeries.hf()
rk2 = BSeries.y() + midpoint_method.b[0] * BSeries.hf() + midpoint_method.b[1] * BSeries.compo_hf(Y1)
series = BSeries.modifying_integrator(rk2)
result = series.sum()
end_time = time.time()
print(result)
print("", end_time - start_time, "seconds")

start_time = time.time()
BSeries.set_order(up_to_order)
Y1 = BSeries.y() + midpoint_method.A[1,0] * BSeries.hf()
rk2 = BSeries.y() + midpoint_method.b[0] * BSeries.hf() + midpoint_method.b[1] * BSeries.compo_hf(Y1)
series = BSeries.modifying_integrator(rk2)
result = series.sum()
end_time = time.time()
print(result)
print("", end_time - start_time, "seconds")

The results are as follows.

Python version 3.9.19 (main, Jul 16 2024, 19:03:54) 
[GCC 11.4.0]

Modified equation
19063/26880
 0.9311249256134033 seconds
19063/26880
 0.915431022644043 seconds

Modifying integrator
5460293/241920
 0.8035497665405273 seconds
5460293/241920
 0.7955782413482666 seconds

This Julia package BSeries.jl

Finally, we perform the same task using BSeries.jl in Julia.

using BSeries, StaticArrays

A = @SArray [0 0; 1//2 0]
b = @SArray [0, 1//1]
c = @SArray [0, 1//2]
up_to_order = 9


println("Modified equation")
@time begin
  series = modified_equation(A, b, c, up_to_order)
  println(sum(values(series)))
end

@time begin
  series = modified_equation(A, b, c, up_to_order)
  println(sum(values(series)))
end


println("\nModifying integrator")
@time begin
  series = modifying_integrator(A, b, c, up_to_order)
  println(sum(values(series)))
end

@time begin
  series = modifying_integrator(A, b, c, up_to_order)
  println(sum(values(series)))
end


println("\nEnergy preservation")
@time begin
  series = bseries(AverageVectorFieldMethod(), up_to_order)
  println(is_energy_preserving(series))
end

@time begin
  series = bseries(AverageVectorFieldMethod(), up_to_order)
  println(is_energy_preserving(series))
end


println("\nSymplecticity (conservation of quadratic invariants)")
@time begin
  # implicit midpoint method = first Gauss method
  A = @SArray [1//2;;]
  b = @SArray [1//1]
  rk = RungeKuttaMethod(A, b)
  series = bseries(rk, up_to_order)
  println(is_symplectic(series))
end

@time begin
  # implicit midpoint method = first Gauss method
  A = @SArray [1//2;;]
  b = @SArray [1//1]
  rk = RungeKuttaMethod(A, b)
  series = bseries(rk, up_to_order)
  println(is_symplectic(series))
end

Modified equation
19063//26880
  0.184782 seconds (71.38 k allocations: 5.656 MiB, 69.04% compilation time)
19063//26880
  0.057088 seconds (591 allocations: 567.703 KiB)

Modifying integrator
5460293//241920
  0.031696 seconds (617 allocations: 534.000 KiB, 12.25% compilation time)
5460293//241920
  0.028800 seconds (576 allocations: 532.578 KiB)

Energy preservation
true
  1.995068 seconds (2.38 M allocations: 164.922 MiB, 1.15% gc time, 96.86% compilation time: 5% of which was recompilation)
true
  0.061200 seconds (38.78 k allocations: 5.073 MiB)

Symplecticity (conservation of quadratic invariants)
true
  0.396870 seconds (260.71 k allocations: 18.064 MiB, 99.80% compilation time)
true
  0.000668 seconds (931 allocations: 325.297 KiB)

References

Hendrik Ranocha and David Ketcheson (2020) Energy Stability of Explicit Runge-Kutta Methods for Nonautonomous or Nonlinear Problems. SIAM Journal on Numerical Analysis DOI: 10.1137/19M1290346