Artificial Intelligence1 min read

Show HN: Automatic deduction of FlashAttention kernel from ~25 lines of SQL

By · Published by Everything Blog

In short

The demonstration relies on an optimizer named einfold, which automatically produces the FlashAttention execution plan from standard relational queries. Inputs are formatted as tables of tokens, features, and values derived from xarray Datasets to compute attention using joins and sums. Rather than hardcoding transformer mechanics, the engine applies a generalized fusion rule for softmax-weighted averages that also optimizes non-attention operations like the Nadaraya–Watson estimator. The system evaluates causal attention across engines such as DuckDB and DataFusion, verifying forward passes and ddx-generated gradient calculations directly against JAX.

Key points

  • Relational query optimizers can reproduce specialized machine learning algorithms without…: Relational query optimizers can reproduce specialized machine learning algorithms without requiring attention-specific logic.
  • Input data structures leverage xarray Datasets to represent attention mechanics as sums c…: Input data structures leverage xarray Datasets to represent attention mechanics as sums calculated across standard relational joins.
  • Forward outputs and gradients produced in a single SQL statement match results from Googl…: Forward outputs and gradients produced in a single SQL statement match results from Google's JAX library.

Sources checked by our analysis pipeline: 1

Repository navigation

Expand file tree

/

Copy path01_attention.py

More file actions

322 lines (282 loc) · 11.7 KB

/

Copy path01_attention.py

File metadata and controls

322 lines (282 loc) · 11.7 KB

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

Artificial Intelligence